2023-01-11T21:02:11.5860909Z Requested labels: linux.8xlarge.nvidia.gpu 2023-01-11T21:02:11.5861015Z Job defined at: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/pull/91627/merge 2023-01-11T21:02:11.5861160Z Reusable workflow chain: 2023-01-11T21:02:11.5861201Z pytorch/pytorch/.github/workflows/pull.yml@refs/pull/91627/merge (57fc38f02f250896a12b32cfa200a6105a03d09c) 2023-01-11T21:02:11.5861251Z -> pytorch/pytorch/.github/workflows/_linux-test.yml@refs/pull/91627/merge (57fc38f02f250896a12b32cfa200a6105a03d09c) 2023-01-11T21:02:11.5861279Z Waiting for a runner to pick up this job... 2023-01-11T21:02:12.2974790Z Job is about to start running on the runner: i-011ccd2732b3aac12 (organization) 2023-01-11T21:02:16.5584840Z Current runner version: '2.300.2' 2023-01-11T21:02:16.5592248Z Runner name: 'i-011ccd2732b3aac12' 2023-01-11T21:02:16.5592956Z Runner group name: 'Default' 2023-01-11T21:02:16.5593671Z Machine name: 'ip-10-0-0-201' 2023-01-11T21:02:16.5596452Z ##[group]GITHUB_TOKEN Permissions 2023-01-11T21:02:16.5597316Z Actions: read 2023-01-11T21:02:16.5597711Z Checks: read 2023-01-11T21:02:16.5598202Z Contents: read 2023-01-11T21:02:16.5598649Z Deployments: read 2023-01-11T21:02:16.5599092Z Discussions: read 2023-01-11T21:02:16.5599471Z Issues: read 2023-01-11T21:02:16.5599927Z Metadata: read 2023-01-11T21:02:16.5600427Z Packages: read 2023-01-11T21:02:16.5600798Z Pages: read 2023-01-11T21:02:16.5601238Z PullRequests: read 2023-01-11T21:02:16.5601722Z RepositoryProjects: read 2023-01-11T21:02:16.5602189Z SecurityEvents: read 2023-01-11T21:02:16.5602631Z Statuses: read 2023-01-11T21:02:16.5603053Z ##[endgroup] 2023-01-11T21:02:16.5607255Z Secret source: None 2023-01-11T21:02:16.5608147Z Prepare workflow directory 2023-01-11T21:02:16.6898674Z Prepare all required actions 2023-01-11T21:02:16.7120906Z Getting action download info 2023-01-11T21:02:17.4999350Z Download action repository 'pytorch/test-infra@main' (SHA:2c225610d00fb13c04fcd60389d3e4d8326167c3) 2023-01-11T21:02:17.8464586Z Download action repository 'pytorch/pytorch@master' (SHA:c5836153f5332ca83d5cacde38f2829a4d54793e) 2023-01-11T21:02:21.1117778Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2023-01-11T21:02:21.5731709Z Getting action download info 2023-01-11T21:02:21.7654228Z Download action repository 'malfet/checkout@silent-checkout' (SHA:c7b8fef48edfe1bca0044a44b1f7f7c4318a3076) 2023-01-11T21:02:21.9780017Z Getting action download info 2023-01-11T21:02:22.2100169Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2023-01-11T21:02:22.3825633Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml 2023-01-11T21:02:22.3828053Z ##[group] Inputs 2023-01-11T21:02:22.3828431Z build-environment: linux-bionic-cuda11.6-py3.10-gcc7 2023-01-11T21:02:22.3829699Z test-matrix: { include: [ { config: "default", shard: 1, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" }, { config: "default", shard: 2, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" }, { config: "default", shard: 3, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" }, { config: "default", shard: 4, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" }, { config: "distributed", shard: 1, num_shards: 3, runner: "linux.8xlarge.nvidia.gpu" }, { config: "distributed", shard: 2, num_shards: 3, runner: "linux.8xlarge.nvidia.gpu" }, { config: "distributed", shard: 3, num_shards: 3, runner: "linux.8xlarge.nvidia.gpu" }, { config: "functorch", shard: 1, num_shards: 1, runner: "linux.4xlarge.nvidia.gpu" }, { config: "deploy", shard: 1, num_shards: 1, runner: "linux.4xlarge.nvidia.gpu" }, ]} 2023-01-11T21:02:22.3831046Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd 2023-01-11T21:02:22.3831546Z sync-tag: 2023-01-11T21:02:22.3832598Z timeout-minutes: 240 2023-01-11T21:02:22.3832868Z use-gha: 2023-01-11T21:02:22.3833139Z ##[endgroup] 2023-01-11T21:02:22.3833954Z Complete job name: linux-bionic-cuda11.6-py3.10-gcc7 / test (distributed, 1, 3, linux.8xlarge.nvidia.gpu) 2023-01-11T21:02:22.4908733Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2023-01-11T21:02:22.4909121Z with: 2023-01-11T21:02:22.4909689Z github-secret: *** 2023-01-11T21:02:22.4910177Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2023-01-11T21:02:22.4910636Z activate-with-label: false 2023-01-11T21:02:22.4910910Z label: with-ssh 2023-01-11T21:02:22.4911159Z remove-existing-keys: true 2023-01-11T21:02:22.4911418Z env: 2023-01-11T21:02:22.4911669Z GIT_DEFAULT_BRANCH: master 2023-01-11T21:02:22.4912080Z ##[endgroup] 2023-01-11T21:02:23.2024095Z Grabbing public ssh keys from https://github.com/LucaLumetti.keys 2023-01-11T21:02:23.2852458Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2023-01-11T21:02:23.2873237Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2023-01-11T21:02:23.2918009Z Login using: ssh ec2-user@ec2-3-86-2-225.compute-1.amazonaws.com 2023-01-11T21:02:23.2918460Z All testing is done inside the container, to start an interactive session run: 2023-01-11T21:02:23.2918927Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2023-01-11T21:02:23.3200165Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@master 2023-01-11T21:02:23.3200537Z with: 2023-01-11T21:02:23.3200773Z submodules: recursive 2023-01-11T21:02:23.3201008Z fetch-depth: 0 2023-01-11T21:02:23.3201236Z env: 2023-01-11T21:02:23.3201470Z GIT_DEFAULT_BRANCH: master 2023-01-11T21:02:23.3201708Z ##[endgroup] 2023-01-11T21:02:23.3487360Z ##[group]Run retry () { 2023-01-11T21:02:23.3487676Z retry () { 2023-01-11T21:02:23.3487981Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2023-01-11T21:02:23.3488252Z } 2023-01-11T21:02:23.3488505Z echo "${GITHUB_WORKSPACE}" 2023-01-11T21:02:23.3488795Z if [ -z "${NO_SUDO}" ]; then 2023-01-11T21:02:23.3489083Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2023-01-11T21:02:23.3489358Z else 2023-01-11T21:02:23.3489624Z  retry rm -rf "${GITHUB_WORKSPACE}" 2023-01-11T21:02:23.3489868Z fi 2023-01-11T21:02:23.3490159Z mkdir "${GITHUB_WORKSPACE}" 2023-01-11T21:02:23.3508772Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-01-11T21:02:23.3509083Z env: 2023-01-11T21:02:23.3509332Z GIT_DEFAULT_BRANCH: master 2023-01-11T21:02:23.3509585Z NO_SUDO: 2023-01-11T21:02:23.3509807Z ##[endgroup] 2023-01-11T21:02:23.3634460Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2023-01-11T21:02:23.3998081Z ##[group]Run malfet/checkout@silent-checkout 2023-01-11T21:02:23.3998352Z with: 2023-01-11T21:02:23.3998628Z ref: 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e 2023-01-11T21:02:23.3998913Z fetch-depth: 0 2023-01-11T21:02:23.3999144Z submodules: recursive 2023-01-11T21:02:23.3999401Z quiet-checkout: true 2023-01-11T21:02:23.3999671Z repository: pytorch/pytorch 2023-01-11T21:02:23.4000098Z token: *** 2023-01-11T21:02:23.4000349Z ssh-strict: true 2023-01-11T21:02:23.4000754Z persist-credentials: true 2023-01-11T21:02:23.4001018Z clean: true 2023-01-11T21:02:23.4001236Z lfs: false 2023-01-11T21:02:23.4001484Z set-safe-directory: true 2023-01-11T21:02:23.4001728Z env: 2023-01-11T21:02:23.4001945Z GIT_DEFAULT_BRANCH: master 2023-01-11T21:02:23.4002198Z ##[endgroup] 2023-01-11T21:02:23.5487894Z Syncing repository: pytorch/pytorch 2023-01-11T21:02:23.5489892Z ##[group]Getting Git version info 2023-01-11T21:02:23.5490833Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2023-01-11T21:02:23.5491794Z [command]/usr/bin/git version 2023-01-11T21:02:23.5492435Z git version 2.38.1 2023-01-11T21:02:23.5494062Z ##[endgroup] 2023-01-11T21:02:23.5512952Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/1f326b1e-3daa-4ecb-baff-c4c1826601bb' before making global git config changes 2023-01-11T21:02:23.5514678Z Adding repository directory to the temporary git global config as a safe directory 2023-01-11T21:02:23.5520524Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2023-01-11T21:02:23.5565169Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2023-01-11T21:02:23.5572065Z ##[group]Initializing the repository 2023-01-11T21:02:23.5575741Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2023-01-11T21:02:23.5605913Z hint: Using 'master' as the name for the initial branch. This default branch name 2023-01-11T21:02:23.5607345Z hint: is subject to change. To configure the initial branch name to use in all 2023-01-11T21:02:23.5608201Z hint: of your new repositories, which will suppress this warning, call: 2023-01-11T21:02:23.5608594Z hint: 2023-01-11T21:02:23.5609145Z hint: git config --global init.defaultBranch 2023-01-11T21:02:23.5609887Z hint: 2023-01-11T21:02:23.5611018Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2023-01-11T21:02:23.5611545Z hint: 'development'. The just-created branch can be renamed via this command: 2023-01-11T21:02:23.5611931Z hint: 2023-01-11T21:02:23.5612426Z hint: git branch -m 2023-01-11T21:02:23.5612960Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2023-01-11T21:02:23.5620314Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2023-01-11T21:02:23.5653827Z ##[endgroup] 2023-01-11T21:02:23.5654825Z ##[group]Disabling automatic garbage collection 2023-01-11T21:02:23.5658415Z [command]/usr/bin/git config --local gc.auto 0 2023-01-11T21:02:23.5688023Z ##[endgroup] 2023-01-11T21:02:23.5689354Z ##[group]Setting up auth 2023-01-11T21:02:23.5698759Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2023-01-11T21:02:23.5731338Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || : 2023-01-11T21:02:23.6067283Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2023-01-11T21:02:23.6097506Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || : 2023-01-11T21:02:23.6390478Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2023-01-11T21:02:23.6441953Z ##[endgroup] 2023-01-11T21:02:23.6442543Z ##[group]Fetching the repository 2023-01-11T21:02:23.6451335Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --quiet --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2023-01-11T21:03:19.9675422Z [command]/usr/bin/git rev-parse --verify --quiet 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e^{object} 2023-01-11T21:03:19.9701733Z 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e 2023-01-11T21:03:19.9707757Z ##[endgroup] 2023-01-11T21:03:19.9708254Z ##[group]Determining the checkout info 2023-01-11T21:03:19.9709641Z ##[endgroup] 2023-01-11T21:03:19.9710073Z ##[group]Checking out the ref 2023-01-11T21:03:19.9715087Z [command]/usr/bin/git checkout --quiet --force 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e 2023-01-11T21:03:21.6956859Z ##[endgroup] 2023-01-11T21:03:21.6957555Z ##[group]Setting up auth for fetching submodules 2023-01-11T21:03:21.6963292Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2023-01-11T21:03:21.7014986Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2023-01-11T21:03:21.7046027Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2023-01-11T21:03:21.7077136Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2023-01-11T21:03:21.7107645Z ##[endgroup] 2023-01-11T21:03:21.7108213Z ##[group]Fetching submodules 2023-01-11T21:03:21.7113189Z [command]/usr/bin/git submodule sync --recursive 2023-01-11T21:03:21.7429770Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2023-01-11T21:03:21.7727354Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2023-01-11T21:03:21.7729819Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2023-01-11T21:03:21.7733231Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2023-01-11T21:03:21.7736427Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2023-01-11T21:03:21.7739704Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK' 2023-01-11T21:03:21.7743253Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2023-01-11T21:03:21.7746775Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2023-01-11T21:03:21.7750431Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2023-01-11T21:03:21.7754164Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2023-01-11T21:03:21.7758002Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub' 2023-01-11T21:03:21.7762285Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2023-01-11T21:03:21.7766301Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2023-01-11T21:03:21.7770958Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2023-01-11T21:03:21.7775734Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2023-01-11T21:03:21.7780328Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2023-01-11T21:03:21.7784925Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2023-01-11T21:03:21.7789784Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2023-01-11T21:03:21.7794835Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2023-01-11T21:03:21.7799827Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2023-01-11T21:03:21.7805218Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2023-01-11T21:03:21.7810867Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2023-01-11T21:03:21.7817695Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake' 2023-01-11T21:03:21.7823307Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2023-01-11T21:03:21.7829030Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2023-01-11T21:03:21.7835014Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2023-01-11T21:03:21.7840909Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse' 2023-01-11T21:03:21.7847100Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2023-01-11T21:03:21.7853797Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2023-01-11T21:03:21.7860270Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt' 2023-01-11T21:03:21.7866867Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2023-01-11T21:03:21.7873356Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2023-01-11T21:03:21.7880271Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2023-01-11T21:03:21.7887081Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2023-01-11T21:03:21.7894583Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2023-01-11T21:03:21.7901841Z Submodule 'third_party/python-enum' (https://github.com/PeachPy/enum34.git) registered for path 'third_party/python-enum' 2023-01-11T21:03:21.7909036Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2023-01-11T21:03:21.7916392Z Submodule 'third_party/python-six' (https://github.com/benjaminp/six.git) registered for path 'third_party/python-six' 2023-01-11T21:03:21.7923749Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2023-01-11T21:03:21.7931644Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb' 2023-01-11T21:03:21.7939905Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2023-01-11T21:03:21.7947823Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd' 2023-01-11T21:03:21.7977056Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2023-01-11T21:03:22.1139665Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2023-01-11T21:03:22.3624338Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2023-01-11T21:03:22.5815149Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2023-01-11T21:03:22.9161198Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'... 2023-01-11T21:03:23.2163936Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2023-01-11T21:03:25.3418607Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2023-01-11T21:03:31.2532608Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2023-01-11T21:03:31.7027266Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2023-01-11T21:03:32.2766361Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cub'... 2023-01-11T21:03:33.8010835Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2023-01-11T21:03:35.1649762Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2023-01-11T21:03:36.5641935Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2023-01-11T21:03:43.3615672Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2023-01-11T21:03:44.1789165Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2023-01-11T21:03:45.8069772Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2023-01-11T21:03:46.9925758Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2023-01-11T21:03:47.2405772Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2023-01-11T21:03:47.7798238Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2023-01-11T21:03:48.1431109Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2023-01-11T21:03:49.1965039Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2023-01-11T21:03:49.6628330Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'... 2023-01-11T21:03:49.9010073Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2023-01-11T21:03:50.1942548Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2023-01-11T21:03:51.8968690Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2023-01-11T21:03:52.2705482Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'... 2023-01-11T21:03:52.7100550Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2023-01-11T21:03:58.7835086Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2023-01-11T21:04:00.5872453Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'... 2023-01-11T21:04:01.1123529Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2023-01-11T21:04:01.4064782Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2023-01-11T21:04:09.0557767Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2023-01-11T21:04:09.2789200Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2023-01-11T21:04:09.5897982Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2023-01-11T21:04:10.5269469Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-enum'... 2023-01-11T21:04:10.7856134Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2023-01-11T21:04:11.1494417Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-six'... 2023-01-11T21:04:11.6857130Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2023-01-11T21:04:12.8633856Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tbb'... 2023-01-11T21:04:15.4414519Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2023-01-11T21:04:15.9774085Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/zstd'... 2023-01-11T21:04:18.9012608Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2023-01-11T21:04:18.9135609Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2023-01-11T21:04:18.9231769Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2023-01-11T21:04:18.9506881Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2023-01-11T21:04:18.9781561Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c' 2023-01-11T21:04:19.0206656Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2023-01-11T21:04:19.7669822Z Submodule path 'third_party/XNNPACK': checked out 'ae108ef49aa5623b896fc93d4298c49d1750d9ba' 2023-01-11T21:04:19.7916320Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2023-01-11T21:04:19.9138014Z Submodule path 'third_party/cpuinfo': checked out '8ec7bd91ad0470e61cf38f618cc1f270dede599c' 2023-01-11T21:04:19.9529362Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4' 2023-01-11T21:04:20.3063495Z Submodule path 'third_party/cudnn_frontend': checked out '171a7a986f7fbd9ed71bd0cf3c7ad4f55843d6b3' 2023-01-11T21:04:20.8251970Z Submodule path 'third_party/cutlass': checked out 'b72cbf957df8cf84a6d0ff91c190ad51a9c1d24a' 2023-01-11T21:04:21.1256477Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2023-01-11T21:04:21.1828232Z Submodule path 'third_party/fbgemm': checked out '80d64206c07879fd4683be66873de7cefa1a0a71' 2023-01-11T21:04:21.1845970Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2023-01-11T21:04:21.1849124Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2023-01-11T21:04:21.1852739Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2023-01-11T21:04:21.1856280Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2023-01-11T21:04:21.1883578Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2023-01-11T21:04:22.1911253Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2023-01-11T21:04:22.7726537Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2023-01-11T21:04:23.8779965Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2023-01-11T21:04:24.2356835Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2023-01-11T21:04:24.3573299Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2023-01-11T21:04:24.4258648Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2023-01-11T21:04:24.4372417Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '1840658c184f3eeba787dae0f06c45756c1daaf5' 2023-01-11T21:04:24.5407178Z Submodule path 'third_party/flatbuffers': checked out 'd0cede9c90c5257537c293517a21376408b549fa' 2023-01-11T21:04:24.5823552Z Submodule path 'third_party/fmt': checked out '7bdf0628b1276379886c7f6dda2cef2b3b374f0b' 2023-01-11T21:04:24.5921529Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2023-01-11T21:04:24.6383237Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2023-01-11T21:04:24.6659460Z Submodule path 'third_party/gloo': checked out '4a5e339b764261d20fc409071dc7a8b8989aa195' 2023-01-11T21:04:24.7185441Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2023-01-11T21:04:24.7313843Z Submodule path 'third_party/ideep': checked out 'e533c771a1e75a1c225c14b2261eefa62681d9e6' 2023-01-11T21:04:24.7330487Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2023-01-11T21:04:24.7356473Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2023-01-11T21:04:33.4605176Z Submodule path 'third_party/ideep/mkl-dnn': checked out '404ad76ee633c939d705eb583ffe50a806969d5e' 2023-01-11T21:04:33.4625341Z Submodule 'third_party/oneDNN' (https://github.com/oneapi-src/oneDNN.git) registered for path 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2023-01-11T21:04:33.4652084Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn/third_party/oneDNN'... 2023-01-11T21:04:42.4412860Z Submodule path 'third_party/ideep/mkl-dnn/third_party/oneDNN': checked out 'fbec3e25a559ee252022ae066817b204e106a6ba' 2023-01-11T21:04:42.4530111Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724' 2023-01-11T21:04:42.4698262Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2023-01-11T21:04:42.5818213Z Submodule path 'third_party/kineto': checked out '6c1629809068efd78a8d56b4aa479c7ec49ae562' 2023-01-11T21:04:42.5835749Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2023-01-11T21:04:42.5839138Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2023-01-11T21:04:42.5866145Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2023-01-11T21:04:43.7748745Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2023-01-11T21:04:44.8271993Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '2591ab91c3898c9f6544fff04660276537d32ffd' 2023-01-11T21:04:44.8908892Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2023-01-11T21:04:44.9159670Z Submodule path 'third_party/nccl/nccl': checked out 'f89fd4777d2ef9229c039ff750ae21da01626f52' 2023-01-11T21:04:44.9312078Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a' 2023-01-11T21:04:45.0618725Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2023-01-11T21:04:45.3919793Z Submodule path 'third_party/onnx': checked out 'f7ee1ac60d06abe8e26c9b6bbe1e3db5286b614b' 2023-01-11T21:04:45.3951581Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2023-01-11T21:04:45.3954690Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2023-01-11T21:04:45.3982345Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2023-01-11T21:04:45.8314461Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2023-01-11T21:04:46.7884781Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2023-01-11T21:04:46.8251446Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'ffa346860b306c9bbfb341aed9c14c067751feb8' 2023-01-11T21:04:46.8424318Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f' 2023-01-11T21:04:46.8441707Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx' 2023-01-11T21:04:46.8467877Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'... 2023-01-11T21:04:48.8204391Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8' 2023-01-11T21:04:48.8227077Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2023-01-11T21:04:48.8230189Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2023-01-11T21:04:48.8257917Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'... 2023-01-11T21:04:49.2795056Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'... 2023-01-11T21:04:50.2111705Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508' 2023-01-11T21:04:50.2904296Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c' 2023-01-11T21:04:50.2921643Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2023-01-11T21:04:50.2948959Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'... 2023-01-11T21:04:50.6085773Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2023-01-11T21:04:50.6187105Z Submodule path 'third_party/pocketfft': checked out 'ea778e37710c07723435b1be58235996d1d43a5a' 2023-01-11T21:04:50.9409859Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2023-01-11T21:04:50.9432872Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2023-01-11T21:04:50.9436314Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2023-01-11T21:04:50.9464240Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2023-01-11T21:04:51.4020838Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2023-01-11T21:04:53.5920886Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2023-01-11T21:04:53.6749601Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2023-01-11T21:04:53.6843962Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2023-01-11T21:04:53.6968347Z Submodule path 'third_party/pthreadpool': checked out 'a134dd5d4cee80cce15db81a72e7f929d71dd413' 2023-01-11T21:04:53.7361146Z Submodule path 'third_party/pybind11': checked out '80dc998efced8ceb2be59756668a7e90e8bef917' 2023-01-11T21:04:53.7460963Z Submodule path 'third_party/python-enum': checked out '4cfedc426c4e2fc52e3f5c2b4297e15ed8d6b8c7' 2023-01-11T21:04:53.7790865Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2023-01-11T21:04:53.7892181Z Submodule path 'third_party/python-six': checked out '15e31431af97e5e64b80af0a3f598d382bcdd49a' 2023-01-11T21:04:53.8418949Z Submodule path 'third_party/sleef': checked out 'e0a003ee838b75d11763aa9c3ef17bf71a725bff' 2023-01-11T21:04:53.9776883Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9' 2023-01-11T21:04:54.0075543Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2023-01-11T21:04:54.0093985Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2023-01-11T21:04:54.0097186Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2023-01-11T21:04:54.0100485Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2023-01-11T21:04:54.0103920Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2023-01-11T21:04:54.0131397Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2023-01-11T21:04:55.1569939Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2023-01-11T21:04:55.4635744Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2023-01-11T21:04:56.7679529Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2023-01-11T21:04:57.7765483Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2023-01-11T21:04:57.7934075Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2023-01-11T21:04:57.8679137Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2023-01-11T21:04:57.9003805Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2023-01-11T21:04:57.9021151Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2023-01-11T21:04:57.9046527Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2023-01-11T21:04:58.1500200Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2023-01-11T21:04:58.3140274Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8' 2023-01-11T21:04:58.3171350Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2023-01-11T21:04:58.3494768Z Entering 'android/libs/fbjni' 2023-01-11T21:04:58.3538115Z Entering 'third_party/FP16' 2023-01-11T21:04:58.3580640Z Entering 'third_party/FXdiv' 2023-01-11T21:04:58.3623316Z Entering 'third_party/NNPACK' 2023-01-11T21:04:58.3666160Z Entering 'third_party/QNNPACK' 2023-01-11T21:04:58.3708545Z Entering 'third_party/VulkanMemoryAllocator' 2023-01-11T21:04:58.3751505Z Entering 'third_party/XNNPACK' 2023-01-11T21:04:58.3805267Z Entering 'third_party/benchmark' 2023-01-11T21:04:58.3847427Z Entering 'third_party/cpuinfo' 2023-01-11T21:04:58.3891179Z Entering 'third_party/cub' 2023-01-11T21:04:58.3935044Z Entering 'third_party/cudnn_frontend' 2023-01-11T21:04:58.3983335Z Entering 'third_party/cutlass' 2023-01-11T21:04:58.4032623Z Entering 'third_party/eigen' 2023-01-11T21:04:58.4077582Z Entering 'third_party/fbgemm' 2023-01-11T21:04:58.4122242Z Entering 'third_party/fbgemm/third_party/asmjit' 2023-01-11T21:04:58.4163056Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2023-01-11T21:04:58.4206591Z Entering 'third_party/fbgemm/third_party/googletest' 2023-01-11T21:04:58.4247915Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2023-01-11T21:04:58.4291538Z Entering 'third_party/flatbuffers' 2023-01-11T21:04:58.4336085Z Entering 'third_party/fmt' 2023-01-11T21:04:58.4377356Z Entering 'third_party/foxi' 2023-01-11T21:04:58.4419210Z Entering 'third_party/gemmlowp/gemmlowp' 2023-01-11T21:04:58.4460725Z Entering 'third_party/gloo' 2023-01-11T21:04:58.4503502Z Entering 'third_party/googletest' 2023-01-11T21:04:58.4544869Z Entering 'third_party/ideep' 2023-01-11T21:04:58.4586452Z Entering 'third_party/ideep/mkl-dnn' 2023-01-11T21:04:58.4629784Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2023-01-11T21:04:58.4677781Z Entering 'third_party/ios-cmake' 2023-01-11T21:04:58.4719162Z Entering 'third_party/ittapi' 2023-01-11T21:04:58.4760331Z Entering 'third_party/kineto' 2023-01-11T21:04:58.4802878Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2023-01-11T21:04:58.4844267Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2023-01-11T21:04:58.4888119Z Entering 'third_party/nccl/nccl' 2023-01-11T21:04:58.4931307Z Entering 'third_party/neon2sse' 2023-01-11T21:04:58.4972798Z Entering 'third_party/nlohmann' 2023-01-11T21:04:58.5016992Z Entering 'third_party/onnx' 2023-01-11T21:04:58.5071819Z Entering 'third_party/onnx/third_party/benchmark' 2023-01-11T21:04:58.5114220Z Entering 'third_party/onnx/third_party/pybind11' 2023-01-11T21:04:58.5159356Z Entering 'third_party/onnx-tensorrt' 2023-01-11T21:04:58.5201048Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2023-01-11T21:04:58.5248054Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2023-01-11T21:04:58.5289718Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2023-01-11T21:04:58.5332201Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2023-01-11T21:04:58.5379068Z Entering 'third_party/pocketfft' 2023-01-11T21:04:58.5420867Z Entering 'third_party/protobuf' 2023-01-11T21:04:58.5466346Z Entering 'third_party/protobuf/third_party/benchmark' 2023-01-11T21:04:58.5508084Z Entering 'third_party/protobuf/third_party/googletest' 2023-01-11T21:04:58.5552702Z Entering 'third_party/psimd' 2023-01-11T21:04:58.5595104Z Entering 'third_party/pthreadpool' 2023-01-11T21:04:58.5637333Z Entering 'third_party/pybind11' 2023-01-11T21:04:58.5679837Z Entering 'third_party/python-enum' 2023-01-11T21:04:58.5721926Z Entering 'third_party/python-peachpy' 2023-01-11T21:04:58.5763677Z Entering 'third_party/python-six' 2023-01-11T21:04:58.5805175Z Entering 'third_party/sleef' 2023-01-11T21:04:58.5846913Z Entering 'third_party/tbb' 2023-01-11T21:04:58.5891719Z Entering 'third_party/tensorpipe' 2023-01-11T21:04:58.5934096Z Entering 'third_party/tensorpipe/third_party/googletest' 2023-01-11T21:04:58.5976557Z Entering 'third_party/tensorpipe/third_party/libnop' 2023-01-11T21:04:58.6019500Z Entering 'third_party/tensorpipe/third_party/libuv' 2023-01-11T21:04:58.6061477Z Entering 'third_party/tensorpipe/third_party/pybind11' 2023-01-11T21:04:58.6103290Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2023-01-11T21:04:58.6147145Z Entering 'third_party/zstd' 2023-01-11T21:04:58.6201770Z ##[endgroup] 2023-01-11T21:04:58.6202280Z ##[group]Persisting credentials for submodules 2023-01-11T21:04:58.6209647Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || : 2023-01-11T21:04:58.6520982Z Entering 'android/libs/fbjni' 2023-01-11T21:04:58.6563579Z Entering 'third_party/FP16' 2023-01-11T21:04:58.6606168Z Entering 'third_party/FXdiv' 2023-01-11T21:04:58.6648239Z Entering 'third_party/NNPACK' 2023-01-11T21:04:58.6690026Z Entering 'third_party/QNNPACK' 2023-01-11T21:04:58.6732103Z Entering 'third_party/VulkanMemoryAllocator' 2023-01-11T21:04:58.6776035Z Entering 'third_party/XNNPACK' 2023-01-11T21:04:58.6829162Z Entering 'third_party/benchmark' 2023-01-11T21:04:58.6870547Z Entering 'third_party/cpuinfo' 2023-01-11T21:04:58.6912773Z Entering 'third_party/cub' 2023-01-11T21:04:58.6955178Z Entering 'third_party/cudnn_frontend' 2023-01-11T21:04:58.7005060Z Entering 'third_party/cutlass' 2023-01-11T21:04:58.7055389Z Entering 'third_party/eigen' 2023-01-11T21:04:58.7100363Z Entering 'third_party/fbgemm' 2023-01-11T21:04:58.7143852Z Entering 'third_party/fbgemm/third_party/asmjit' 2023-01-11T21:04:58.7186639Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2023-01-11T21:04:58.7228010Z Entering 'third_party/fbgemm/third_party/googletest' 2023-01-11T21:04:58.7270167Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2023-01-11T21:04:58.7312716Z Entering 'third_party/flatbuffers' 2023-01-11T21:04:58.7357604Z Entering 'third_party/fmt' 2023-01-11T21:04:58.7400059Z Entering 'third_party/foxi' 2023-01-11T21:04:58.7443074Z Entering 'third_party/gemmlowp/gemmlowp' 2023-01-11T21:04:58.7485400Z Entering 'third_party/gloo' 2023-01-11T21:04:58.7527285Z Entering 'third_party/googletest' 2023-01-11T21:04:58.7569186Z Entering 'third_party/ideep' 2023-01-11T21:04:58.7609751Z Entering 'third_party/ideep/mkl-dnn' 2023-01-11T21:04:58.7654425Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2023-01-11T21:04:58.7704452Z Entering 'third_party/ios-cmake' 2023-01-11T21:04:58.7747652Z Entering 'third_party/ittapi' 2023-01-11T21:04:58.7791601Z Entering 'third_party/kineto' 2023-01-11T21:04:58.7833657Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2023-01-11T21:04:58.7876391Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2023-01-11T21:04:58.7918992Z Entering 'third_party/nccl/nccl' 2023-01-11T21:04:58.7961456Z Entering 'third_party/neon2sse' 2023-01-11T21:04:58.8003906Z Entering 'third_party/nlohmann' 2023-01-11T21:04:58.8047028Z Entering 'third_party/onnx' 2023-01-11T21:04:58.8101650Z Entering 'third_party/onnx/third_party/benchmark' 2023-01-11T21:04:58.8144594Z Entering 'third_party/onnx/third_party/pybind11' 2023-01-11T21:04:58.8187785Z Entering 'third_party/onnx-tensorrt' 2023-01-11T21:04:58.8228528Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2023-01-11T21:04:58.8275140Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2023-01-11T21:04:58.8317891Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2023-01-11T21:04:58.8359045Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2023-01-11T21:04:58.8406098Z Entering 'third_party/pocketfft' 2023-01-11T21:04:58.8447068Z Entering 'third_party/protobuf' 2023-01-11T21:04:58.8493230Z Entering 'third_party/protobuf/third_party/benchmark' 2023-01-11T21:04:58.8534122Z Entering 'third_party/protobuf/third_party/googletest' 2023-01-11T21:04:58.8577132Z Entering 'third_party/psimd' 2023-01-11T21:04:58.8619868Z Entering 'third_party/pthreadpool' 2023-01-11T21:04:58.8661224Z Entering 'third_party/pybind11' 2023-01-11T21:04:58.8703784Z Entering 'third_party/python-enum' 2023-01-11T21:04:58.8745578Z Entering 'third_party/python-peachpy' 2023-01-11T21:04:58.8786453Z Entering 'third_party/python-six' 2023-01-11T21:04:58.8828590Z Entering 'third_party/sleef' 2023-01-11T21:04:58.8871472Z Entering 'third_party/tbb' 2023-01-11T21:04:58.8914643Z Entering 'third_party/tensorpipe' 2023-01-11T21:04:58.8957117Z Entering 'third_party/tensorpipe/third_party/googletest' 2023-01-11T21:04:58.8997647Z Entering 'third_party/tensorpipe/third_party/libnop' 2023-01-11T21:04:58.9039041Z Entering 'third_party/tensorpipe/third_party/libuv' 2023-01-11T21:04:58.9079969Z Entering 'third_party/tensorpipe/third_party/pybind11' 2023-01-11T21:04:58.9119923Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2023-01-11T21:04:58.9164502Z Entering 'third_party/zstd' 2023-01-11T21:04:58.9220281Z [command]/usr/bin/git submodule foreach --recursive git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url 2023-01-11T21:04:58.9529418Z Entering 'android/libs/fbjni' 2023-01-11T21:04:58.9567068Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2023-01-11T21:04:58.9586441Z Entering 'third_party/FP16' 2023-01-11T21:04:58.9625223Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2023-01-11T21:04:58.9641778Z Entering 'third_party/FXdiv' 2023-01-11T21:04:58.9680544Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2023-01-11T21:04:58.9697726Z Entering 'third_party/NNPACK' 2023-01-11T21:04:58.9735959Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2023-01-11T21:04:58.9753413Z Entering 'third_party/QNNPACK' 2023-01-11T21:04:58.9791927Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config remote.origin.url 2023-01-11T21:04:58.9809002Z Entering 'third_party/VulkanMemoryAllocator' 2023-01-11T21:04:58.9847268Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2023-01-11T21:04:58.9865092Z Entering 'third_party/XNNPACK' 2023-01-11T21:04:58.9903018Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2023-01-11T21:04:58.9930979Z Entering 'third_party/benchmark' 2023-01-11T21:04:58.9968789Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2023-01-11T21:04:58.9986474Z Entering 'third_party/cpuinfo' 2023-01-11T21:04:59.0024683Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2023-01-11T21:04:59.0042684Z Entering 'third_party/cub' 2023-01-11T21:04:59.0081050Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config remote.origin.url 2023-01-11T21:04:59.0098583Z Entering 'third_party/cudnn_frontend' 2023-01-11T21:04:59.0136723Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2023-01-11T21:04:59.0159114Z Entering 'third_party/cutlass' 2023-01-11T21:04:59.0198681Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2023-01-11T21:04:59.0222971Z Entering 'third_party/eigen' 2023-01-11T21:04:59.0260974Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2023-01-11T21:04:59.0280190Z Entering 'third_party/fbgemm' 2023-01-11T21:04:59.0319817Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2023-01-11T21:04:59.0337006Z Entering 'third_party/fbgemm/third_party/asmjit' 2023-01-11T21:04:59.0375046Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2023-01-11T21:04:59.0391908Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2023-01-11T21:04:59.0429777Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2023-01-11T21:04:59.0446841Z Entering 'third_party/fbgemm/third_party/googletest' 2023-01-11T21:04:59.0485703Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2023-01-11T21:04:59.0504720Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2023-01-11T21:04:59.0542478Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2023-01-11T21:04:59.0560283Z Entering 'third_party/flatbuffers' 2023-01-11T21:04:59.0598303Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2023-01-11T21:04:59.0618695Z Entering 'third_party/fmt' 2023-01-11T21:04:59.0657631Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2023-01-11T21:04:59.0674282Z Entering 'third_party/foxi' 2023-01-11T21:04:59.0712255Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2023-01-11T21:04:59.0728970Z Entering 'third_party/gemmlowp/gemmlowp' 2023-01-11T21:04:59.0768225Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2023-01-11T21:04:59.0786424Z Entering 'third_party/gloo' 2023-01-11T21:04:59.0824174Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2023-01-11T21:04:59.0840941Z Entering 'third_party/googletest' 2023-01-11T21:04:59.0880402Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2023-01-11T21:04:59.0898067Z Entering 'third_party/ideep' 2023-01-11T21:04:59.0936845Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2023-01-11T21:04:59.0953337Z Entering 'third_party/ideep/mkl-dnn' 2023-01-11T21:04:59.0991547Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2023-01-11T21:04:59.1010935Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2023-01-11T21:04:59.1049900Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/modules/third_party/oneDNN/config remote.origin.url 2023-01-11T21:04:59.1073575Z Entering 'third_party/ios-cmake' 2023-01-11T21:04:59.1112844Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config remote.origin.url 2023-01-11T21:04:59.1130961Z Entering 'third_party/ittapi' 2023-01-11T21:04:59.1169003Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2023-01-11T21:04:59.1186630Z Entering 'third_party/kineto' 2023-01-11T21:04:59.1226118Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2023-01-11T21:04:59.1243584Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2023-01-11T21:04:59.1282505Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2023-01-11T21:04:59.1300862Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2023-01-11T21:04:59.1338791Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2023-01-11T21:04:59.1356766Z Entering 'third_party/nccl/nccl' 2023-01-11T21:04:59.1396651Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2023-01-11T21:04:59.1414335Z Entering 'third_party/neon2sse' 2023-01-11T21:04:59.1452634Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config remote.origin.url 2023-01-11T21:04:59.1469115Z Entering 'third_party/nlohmann' 2023-01-11T21:04:59.1507619Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2023-01-11T21:04:59.1525656Z Entering 'third_party/onnx' 2023-01-11T21:04:59.1564223Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2023-01-11T21:04:59.1594788Z Entering 'third_party/onnx/third_party/benchmark' 2023-01-11T21:04:59.1633137Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2023-01-11T21:04:59.1650335Z Entering 'third_party/onnx/third_party/pybind11' 2023-01-11T21:04:59.1688577Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2023-01-11T21:04:59.1708372Z Entering 'third_party/onnx-tensorrt' 2023-01-11T21:04:59.1748544Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config remote.origin.url 2023-01-11T21:04:59.1764738Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2023-01-11T21:04:59.1804059Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config remote.origin.url 2023-01-11T21:04:59.1827444Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2023-01-11T21:04:59.1868353Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2023-01-11T21:04:59.1886007Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2023-01-11T21:04:59.1926198Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2023-01-11T21:04:59.1943448Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2023-01-11T21:04:59.1982848Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2023-01-11T21:04:59.2005060Z Entering 'third_party/pocketfft' 2023-01-11T21:04:59.2043862Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2023-01-11T21:04:59.2061154Z Entering 'third_party/protobuf' 2023-01-11T21:04:59.2100253Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2023-01-11T21:04:59.2121175Z Entering 'third_party/protobuf/third_party/benchmark' 2023-01-11T21:04:59.2159606Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2023-01-11T21:04:59.2176804Z Entering 'third_party/protobuf/third_party/googletest' 2023-01-11T21:04:59.2215075Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2023-01-11T21:04:59.2233767Z Entering 'third_party/psimd' 2023-01-11T21:04:59.2272633Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2023-01-11T21:04:59.2289425Z Entering 'third_party/pthreadpool' 2023-01-11T21:04:59.2327691Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2023-01-11T21:04:59.2344603Z Entering 'third_party/pybind11' 2023-01-11T21:04:59.2383499Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2023-01-11T21:04:59.2400883Z Entering 'third_party/python-enum' 2023-01-11T21:04:59.2439150Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-enum/config remote.origin.url 2023-01-11T21:04:59.2456737Z Entering 'third_party/python-peachpy' 2023-01-11T21:04:59.2494966Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2023-01-11T21:04:59.2511788Z Entering 'third_party/python-six' 2023-01-11T21:04:59.2550275Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-six/config remote.origin.url 2023-01-11T21:04:59.2566898Z Entering 'third_party/sleef' 2023-01-11T21:04:59.2605158Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2023-01-11T21:04:59.2622966Z Entering 'third_party/tbb' 2023-01-11T21:04:59.2661084Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config remote.origin.url 2023-01-11T21:04:59.2680544Z Entering 'third_party/tensorpipe' 2023-01-11T21:04:59.2720960Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2023-01-11T21:04:59.2738365Z Entering 'third_party/tensorpipe/third_party/googletest' 2023-01-11T21:04:59.2776700Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2023-01-11T21:04:59.2793640Z Entering 'third_party/tensorpipe/third_party/libnop' 2023-01-11T21:04:59.2830885Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2023-01-11T21:04:59.2847120Z Entering 'third_party/tensorpipe/third_party/libuv' 2023-01-11T21:04:59.2884806Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2023-01-11T21:04:59.2902814Z Entering 'third_party/tensorpipe/third_party/pybind11' 2023-01-11T21:04:59.2941220Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2023-01-11T21:04:59.2956865Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2023-01-11T21:04:59.2995690Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2023-01-11T21:04:59.3015330Z Entering 'third_party/zstd' 2023-01-11T21:04:59.3053845Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config remote.origin.url 2023-01-11T21:04:59.3885517Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2023-01-11T21:04:59.4194209Z Entering 'android/libs/fbjni' 2023-01-11T21:04:59.4235949Z Entering 'third_party/FP16' 2023-01-11T21:04:59.4277972Z Entering 'third_party/FXdiv' 2023-01-11T21:04:59.4320272Z Entering 'third_party/NNPACK' 2023-01-11T21:04:59.4363411Z Entering 'third_party/QNNPACK' 2023-01-11T21:04:59.4405971Z Entering 'third_party/VulkanMemoryAllocator' 2023-01-11T21:04:59.4448895Z Entering 'third_party/XNNPACK' 2023-01-11T21:04:59.4504009Z Entering 'third_party/benchmark' 2023-01-11T21:04:59.4545442Z Entering 'third_party/cpuinfo' 2023-01-11T21:04:59.4588648Z Entering 'third_party/cub' 2023-01-11T21:04:59.4631443Z Entering 'third_party/cudnn_frontend' 2023-01-11T21:04:59.4678893Z Entering 'third_party/cutlass' 2023-01-11T21:04:59.4729244Z Entering 'third_party/eigen' 2023-01-11T21:04:59.4775587Z Entering 'third_party/fbgemm' 2023-01-11T21:04:59.4818181Z Entering 'third_party/fbgemm/third_party/asmjit' 2023-01-11T21:04:59.4861034Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2023-01-11T21:04:59.4990078Z Entering 'third_party/fbgemm/third_party/googletest' 2023-01-11T21:04:59.5031890Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2023-01-11T21:04:59.5074577Z Entering 'third_party/flatbuffers' 2023-01-11T21:04:59.5120082Z Entering 'third_party/fmt' 2023-01-11T21:04:59.5162617Z Entering 'third_party/foxi' 2023-01-11T21:04:59.5205444Z Entering 'third_party/gemmlowp/gemmlowp' 2023-01-11T21:04:59.5248567Z Entering 'third_party/gloo' 2023-01-11T21:04:59.5291677Z Entering 'third_party/googletest' 2023-01-11T21:04:59.5334501Z Entering 'third_party/ideep' 2023-01-11T21:04:59.5376291Z Entering 'third_party/ideep/mkl-dnn' 2023-01-11T21:04:59.5420282Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2023-01-11T21:04:59.5470260Z Entering 'third_party/ios-cmake' 2023-01-11T21:04:59.5512762Z Entering 'third_party/ittapi' 2023-01-11T21:04:59.5557034Z Entering 'third_party/kineto' 2023-01-11T21:04:59.5599376Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2023-01-11T21:04:59.5641100Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2023-01-11T21:04:59.5685867Z Entering 'third_party/nccl/nccl' 2023-01-11T21:04:59.5727720Z Entering 'third_party/neon2sse' 2023-01-11T21:04:59.5769987Z Entering 'third_party/nlohmann' 2023-01-11T21:04:59.5814227Z Entering 'third_party/onnx' 2023-01-11T21:04:59.5871361Z Entering 'third_party/onnx/third_party/benchmark' 2023-01-11T21:04:59.5913514Z Entering 'third_party/onnx/third_party/pybind11' 2023-01-11T21:04:59.5957355Z Entering 'third_party/onnx-tensorrt' 2023-01-11T21:04:59.5999525Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2023-01-11T21:04:59.6047259Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2023-01-11T21:04:59.6089263Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2023-01-11T21:04:59.6130934Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2023-01-11T21:04:59.6178236Z Entering 'third_party/pocketfft' 2023-01-11T21:04:59.6220353Z Entering 'third_party/protobuf' 2023-01-11T21:04:59.6265630Z Entering 'third_party/protobuf/third_party/benchmark' 2023-01-11T21:04:59.6308109Z Entering 'third_party/protobuf/third_party/googletest' 2023-01-11T21:04:59.6352429Z Entering 'third_party/psimd' 2023-01-11T21:04:59.6395733Z Entering 'third_party/pthreadpool' 2023-01-11T21:04:59.6438064Z Entering 'third_party/pybind11' 2023-01-11T21:04:59.6481388Z Entering 'third_party/python-enum' 2023-01-11T21:04:59.6523725Z Entering 'third_party/python-peachpy' 2023-01-11T21:04:59.6565931Z Entering 'third_party/python-six' 2023-01-11T21:04:59.6610233Z Entering 'third_party/sleef' 2023-01-11T21:04:59.6653396Z Entering 'third_party/tbb' 2023-01-11T21:04:59.6698023Z Entering 'third_party/tensorpipe' 2023-01-11T21:04:59.6740927Z Entering 'third_party/tensorpipe/third_party/googletest' 2023-01-11T21:04:59.6782253Z Entering 'third_party/tensorpipe/third_party/libnop' 2023-01-11T21:04:59.6822947Z Entering 'third_party/tensorpipe/third_party/libuv' 2023-01-11T21:04:59.6864732Z Entering 'third_party/tensorpipe/third_party/pybind11' 2023-01-11T21:04:59.6905072Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2023-01-11T21:04:59.6949809Z Entering 'third_party/zstd' 2023-01-11T21:04:59.7004902Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2023-01-11T21:04:59.7311484Z Entering 'android/libs/fbjni' 2023-01-11T21:04:59.7353821Z Entering 'third_party/FP16' 2023-01-11T21:04:59.7396117Z Entering 'third_party/FXdiv' 2023-01-11T21:04:59.7437912Z Entering 'third_party/NNPACK' 2023-01-11T21:04:59.7481802Z Entering 'third_party/QNNPACK' 2023-01-11T21:04:59.7524927Z Entering 'third_party/VulkanMemoryAllocator' 2023-01-11T21:04:59.7567004Z Entering 'third_party/XNNPACK' 2023-01-11T21:04:59.7620770Z Entering 'third_party/benchmark' 2023-01-11T21:04:59.7663151Z Entering 'third_party/cpuinfo' 2023-01-11T21:04:59.7705018Z Entering 'third_party/cub' 2023-01-11T21:04:59.7748857Z Entering 'third_party/cudnn_frontend' 2023-01-11T21:04:59.7796250Z Entering 'third_party/cutlass' 2023-01-11T21:04:59.7844707Z Entering 'third_party/eigen' 2023-01-11T21:04:59.7888598Z Entering 'third_party/fbgemm' 2023-01-11T21:04:59.7931856Z Entering 'third_party/fbgemm/third_party/asmjit' 2023-01-11T21:04:59.7973257Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2023-01-11T21:04:59.8016401Z Entering 'third_party/fbgemm/third_party/googletest' 2023-01-11T21:04:59.8058900Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2023-01-11T21:04:59.8101251Z Entering 'third_party/flatbuffers' 2023-01-11T21:04:59.8145894Z Entering 'third_party/fmt' 2023-01-11T21:04:59.8188989Z Entering 'third_party/foxi' 2023-01-11T21:04:59.8230875Z Entering 'third_party/gemmlowp/gemmlowp' 2023-01-11T21:04:59.8274241Z Entering 'third_party/gloo' 2023-01-11T21:04:59.8316273Z Entering 'third_party/googletest' 2023-01-11T21:04:59.8358668Z Entering 'third_party/ideep' 2023-01-11T21:04:59.8399710Z Entering 'third_party/ideep/mkl-dnn' 2023-01-11T21:04:59.8444164Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2023-01-11T21:04:59.8493133Z Entering 'third_party/ios-cmake' 2023-01-11T21:04:59.8534942Z Entering 'third_party/ittapi' 2023-01-11T21:04:59.8576959Z Entering 'third_party/kineto' 2023-01-11T21:04:59.8618957Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2023-01-11T21:04:59.8660968Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2023-01-11T21:04:59.8704141Z Entering 'third_party/nccl/nccl' 2023-01-11T21:04:59.8745252Z Entering 'third_party/neon2sse' 2023-01-11T21:04:59.8786575Z Entering 'third_party/nlohmann' 2023-01-11T21:04:59.8829998Z Entering 'third_party/onnx' 2023-01-11T21:04:59.8885209Z Entering 'third_party/onnx/third_party/benchmark' 2023-01-11T21:04:59.8927126Z Entering 'third_party/onnx/third_party/pybind11' 2023-01-11T21:04:59.8971133Z Entering 'third_party/onnx-tensorrt' 2023-01-11T21:04:59.9013553Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2023-01-11T21:04:59.9059286Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2023-01-11T21:04:59.9101897Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2023-01-11T21:04:59.9144311Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2023-01-11T21:04:59.9191232Z Entering 'third_party/pocketfft' 2023-01-11T21:04:59.9232758Z Entering 'third_party/protobuf' 2023-01-11T21:04:59.9278684Z Entering 'third_party/protobuf/third_party/benchmark' 2023-01-11T21:04:59.9319772Z Entering 'third_party/protobuf/third_party/googletest' 2023-01-11T21:04:59.9364058Z Entering 'third_party/psimd' 2023-01-11T21:04:59.9406073Z Entering 'third_party/pthreadpool' 2023-01-11T21:04:59.9448221Z Entering 'third_party/pybind11' 2023-01-11T21:04:59.9490816Z Entering 'third_party/python-enum' 2023-01-11T21:04:59.9532837Z Entering 'third_party/python-peachpy' 2023-01-11T21:04:59.9575104Z Entering 'third_party/python-six' 2023-01-11T21:04:59.9618066Z Entering 'third_party/sleef' 2023-01-11T21:04:59.9659613Z Entering 'third_party/tbb' 2023-01-11T21:04:59.9704124Z Entering 'third_party/tensorpipe' 2023-01-11T21:04:59.9746406Z Entering 'third_party/tensorpipe/third_party/googletest' 2023-01-11T21:04:59.9788695Z Entering 'third_party/tensorpipe/third_party/libnop' 2023-01-11T21:04:59.9831224Z Entering 'third_party/tensorpipe/third_party/libuv' 2023-01-11T21:04:59.9874412Z Entering 'third_party/tensorpipe/third_party/pybind11' 2023-01-11T21:04:59.9916012Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2023-01-11T21:04:59.9960419Z Entering 'third_party/zstd' 2023-01-11T21:05:00.0012982Z ##[endgroup] 2023-01-11T21:05:00.0056097Z [command]/usr/bin/git log -1 --format='%H' 2023-01-11T21:05:00.0084065Z '8419ddda87c8a47eacc63b54bc7ec98c1f27c26e' 2023-01-11T21:05:00.0235913Z Prepare all required actions 2023-01-11T21:05:00.0269607Z ##[group]Run ./.github/actions/setup-linux 2023-01-11T21:05:00.0269885Z env: 2023-01-11T21:05:00.0270125Z GIT_DEFAULT_BRANCH: master 2023-01-11T21:05:00.0270361Z ##[endgroup] 2023-01-11T21:05:00.0290011Z ##[group]Run set -euo pipefail 2023-01-11T21:05:00.0290725Z set -euo pipefail 2023-01-11T21:05:00.0291035Z function get_ec2_metadata() { 2023-01-11T21:05:00.0291352Z  # Pulled from instance metadata endpoint for EC2 2023-01-11T21:05:00.0291823Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2023-01-11T21:05:00.0292225Z  category=$1 2023-01-11T21:05:00.0292551Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2023-01-11T21:05:00.0292834Z } 2023-01-11T21:05:00.0293105Z echo "ami-id: $(get_ec2_metadata ami-id)" 2023-01-11T21:05:00.0293485Z echo "instance-id: $(get_ec2_metadata instance-id)" 2023-01-11T21:05:00.0293843Z echo "instance-type: $(get_ec2_metadata instance-type)" 2023-01-11T21:05:00.0294177Z echo "system info $(uname -a)" 2023-01-11T21:05:00.0307060Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-01-11T21:05:00.0307334Z env: 2023-01-11T21:05:00.0307572Z GIT_DEFAULT_BRANCH: master 2023-01-11T21:05:00.0307828Z ##[endgroup] 2023-01-11T21:05:00.0408876Z ami-id: ami-096198a0bccc6bad4 2023-01-11T21:05:00.0471561Z instance-id: i-011ccd2732b3aac12 2023-01-11T21:05:00.0534757Z instance-type: g3.8xlarge 2023-01-11T21:05:00.0543028Z system info Linux ip-10-0-0-201.ec2.internal 4.14.252-195.483.amzn2.x86_64 #1 SMP Mon Nov 1 20:58:46 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux 2023-01-11T21:05:00.0563935Z ##[group]Run if systemctl is-active --quiet docker; then 2023-01-11T21:05:00.0564311Z if systemctl is-active --quiet docker; then 2023-01-11T21:05:00.0564625Z  echo "Docker daemon is running..."; 2023-01-11T21:05:00.0564905Z else 2023-01-11T21:05:00.0565223Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2023-01-11T21:05:00.0565529Z fi 2023-01-11T21:05:00.0578239Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-01-11T21:05:00.0578588Z env: 2023-01-11T21:05:00.0578830Z GIT_DEFAULT_BRANCH: master 2023-01-11T21:05:00.0579069Z ##[endgroup] 2023-01-11T21:05:00.0631259Z Docker daemon is running... 2023-01-11T21:05:00.0651988Z ##[group]Run AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") 2023-01-11T21:05:00.0652451Z AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") 2023-01-11T21:05:00.0652830Z retry () { "$@" || (sleep 1 && "$@") || (sleep 2 && "$@") } 2023-01-11T21:05:00.0653323Z retry aws ecr get-login*** "$AWS_DEFAULT_REGION" | docker login --username AWS \ 2023-01-11T21:05:00.0653785Z  --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2023-01-11T21:05:00.0665115Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-01-11T21:05:00.0665409Z env: 2023-01-11T21:05:00.0665647Z GIT_DEFAULT_BRANCH: master 2023-01-11T21:05:00.0665896Z AWS_RETRY_MODE: standard 2023-01-11T21:05:00.0666152Z AWS_MAX_ATTEMPTS: 5 2023-01-11T21:05:00.0666416Z AWS_DEFAULT_REGION: us-east-1 2023-01-11T21:05:00.0666659Z ##[endgroup] 2023-01-11T21:05:00.9936613Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2023-01-11T21:05:00.9937086Z Configure a credential helper to remove this warning. See 2023-01-11T21:05:00.9937615Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2023-01-11T21:05:00.9937872Z 2023-01-11T21:05:00.9938747Z Login Succeeded 2023-01-11T21:05:01.0004616Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2023-01-11T21:05:01.0005013Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2023-01-11T21:05:01.0005511Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2023-01-11T21:05:01.0019008Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-01-11T21:05:01.0019310Z env: 2023-01-11T21:05:01.0019542Z GIT_DEFAULT_BRANCH: master 2023-01-11T21:05:01.0019805Z ##[endgroup] 2023-01-11T21:05:01.0110971Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2023-01-11T21:05:01.0111316Z with: 2023-01-11T21:05:01.0111797Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd 2023-01-11T21:05:01.0112260Z env: 2023-01-11T21:05:01.0112478Z GIT_DEFAULT_BRANCH: master 2023-01-11T21:05:01.0112730Z ##[endgroup] 2023-01-11T21:05:01.0131951Z ##[group]Run retry () { "$@" || (sleep 1 && "$@") || (sleep 2 && "$@") } 2023-01-11T21:05:01.0132309Z retry () { "$@" || (sleep 1 && "$@") || (sleep 2 && "$@") } 2023-01-11T21:05:01.0132664Z # ignore output since only exit code is used for conditional 2023-01-11T21:05:01.0133040Z # only pull docker image if it's not available locally 2023-01-11T21:05:01.0133437Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2023-01-11T21:05:01.0133824Z  retry docker pull "${DOCKER_IMAGE}" 2023-01-11T21:05:01.0134097Z fi 2023-01-11T21:05:01.0145947Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-01-11T21:05:01.0146247Z env: 2023-01-11T21:05:01.0146488Z GIT_DEFAULT_BRANCH: master 2023-01-11T21:05:01.0146984Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd 2023-01-11T21:05:01.0147467Z ##[endgroup] 2023-01-11T21:05:01.2612963Z fd224c2e6c79d7fdec6408da598bf52bc5b201dd: Pulling from pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7 2023-01-11T21:05:01.2613969Z fb668870d8a7: Pulling fs layer 2023-01-11T21:05:01.2614256Z 3dc32ed140fb: Pulling fs layer 2023-01-11T21:05:01.2614505Z 54a1df240516: Pulling fs layer 2023-01-11T21:05:01.2614787Z cf378b3cb3c7: Pulling fs layer 2023-01-11T21:05:01.2615052Z 9b4412378859: Pulling fs layer 2023-01-11T21:05:01.2615314Z 502253a1be21: Pulling fs layer 2023-01-11T21:05:01.2615582Z 5c7dd67e5809: Pulling fs layer 2023-01-11T21:05:01.2615843Z bdfd23ed3f48: Pulling fs layer 2023-01-11T21:05:01.2616122Z aee1dd761bdd: Pulling fs layer 2023-01-11T21:05:01.2616390Z 5feda9af2542: Pulling fs layer 2023-01-11T21:05:01.2616647Z f8371ecb849a: Pulling fs layer 2023-01-11T21:05:01.2616929Z ce4a87d45645: Pulling fs layer 2023-01-11T21:05:01.2617287Z 39629f7269f9: Pulling fs layer 2023-01-11T21:05:01.2617962Z 87d0ffa55850: Pulling fs layer 2023-01-11T21:05:01.2618625Z 70702f8b5bc4: Pulling fs layer 2023-01-11T21:05:01.2619703Z cf378b3cb3c7: Waiting 2023-01-11T21:05:01.2620068Z 0c06be5c20e0: Pulling fs layer 2023-01-11T21:05:01.2620436Z 9b4412378859: Waiting 2023-01-11T21:05:01.2620705Z ce4a87d45645: Waiting 2023-01-11T21:05:01.2621522Z b372c2a3bc3f: Pulling fs layer 2023-01-11T21:05:01.2621862Z 582d081a59fa: Pulling fs layer 2023-01-11T21:05:01.2622157Z e1c655e7ec0e: Pulling fs layer 2023-01-11T21:05:01.2622491Z c7726d39d806: Pulling fs layer 2023-01-11T21:05:01.2622960Z 5c7dd67e5809: Waiting 2023-01-11T21:05:01.2623268Z 1c22f2f8c01b: Pulling fs layer 2023-01-11T21:05:01.2623599Z bdfd23ed3f48: Waiting 2023-01-11T21:05:01.2625511Z b8f759fd0191: Pulling fs layer 2023-01-11T21:05:01.2626011Z e28e73a4bddd: Pulling fs layer 2023-01-11T21:05:01.2626338Z 39629f7269f9: Waiting 2023-01-11T21:05:01.2626700Z 90d8f9bbe048: Pulling fs layer 2023-01-11T21:05:01.2627087Z 5feda9af2542: Waiting 2023-01-11T21:05:01.2627361Z b34bd39d0461: Pulling fs layer 2023-01-11T21:05:01.2627682Z c7726d39d806: Waiting 2023-01-11T21:05:01.2627994Z 1c22f2f8c01b: Waiting 2023-01-11T21:05:01.2628247Z b8f759fd0191: Waiting 2023-01-11T21:05:01.2628626Z 0c06be5c20e0: Waiting 2023-01-11T21:05:01.2628947Z 2f2308643d60: Pulling fs layer 2023-01-11T21:05:01.2629225Z b372c2a3bc3f: Waiting 2023-01-11T21:05:01.2629541Z 8e3432e5a569: Pulling fs layer 2023-01-11T21:05:01.2629866Z e28e73a4bddd: Waiting 2023-01-11T21:05:01.2630157Z 9ea746919509: Pulling fs layer 2023-01-11T21:05:01.2630484Z e1c655e7ec0e: Waiting 2023-01-11T21:05:01.2630791Z 8e3432e5a569: Waiting 2023-01-11T21:05:01.2631158Z 1a2fd7b216d7: Pulling fs layer 2023-01-11T21:05:01.2631549Z 19fde6a723a0: Pulling fs layer 2023-01-11T21:05:01.2631935Z 06369252d749: Pulling fs layer 2023-01-11T21:05:01.2632223Z ea4bfeaa0fc7: Pulling fs layer 2023-01-11T21:05:01.2632562Z a1d16b6a5070: Pulling fs layer 2023-01-11T21:05:01.2632895Z f550b7ff2470: Pulling fs layer 2023-01-11T21:05:01.2633160Z 9ea746919509: Waiting 2023-01-11T21:05:01.2633525Z 12ddc57b99eb: Pulling fs layer 2023-01-11T21:05:01.2633856Z 8345085fb0a0: Pulling fs layer 2023-01-11T21:05:01.2634138Z 4cc94dbec031: Pulling fs layer 2023-01-11T21:05:01.2634489Z a1d16b6a5070: Waiting 2023-01-11T21:05:01.2634844Z ea4bfeaa0fc7: Waiting 2023-01-11T21:05:01.2635108Z f550b7ff2470: Waiting 2023-01-11T21:05:01.2635416Z 12ddc57b99eb: Waiting 2023-01-11T21:05:01.2635721Z f8371ecb849a: Waiting 2023-01-11T21:05:01.2636000Z 29a7c0d5fa4c: Pulling fs layer 2023-01-11T21:05:01.2636351Z 70702f8b5bc4: Waiting 2023-01-11T21:05:01.2636665Z 25571655d0e1: Pulling fs layer 2023-01-11T21:05:01.2676192Z bdf297d7f88c: Pulling fs layer 2023-01-11T21:05:01.2676537Z 29a7c0d5fa4c: Waiting 2023-01-11T21:05:01.2677057Z 0b3950af8ae1: Pulling fs layer 2023-01-11T21:05:01.2677635Z 8345085fb0a0: Waiting 2023-01-11T21:05:01.2678034Z bdf297d7f88c: Waiting 2023-01-11T21:05:01.2678371Z 6d68f7da8baa: Pulling fs layer 2023-01-11T21:05:01.2678749Z cca768f96df4: Pulling fs layer 2023-01-11T21:05:01.2679103Z 8c3cf3d5e1c5: Pulling fs layer 2023-01-11T21:05:01.2679441Z 61eecfa8b34e: Pulling fs layer 2023-01-11T21:05:01.2679776Z 0b3950af8ae1: Waiting 2023-01-11T21:05:01.2680035Z 6d68f7da8baa: Waiting 2023-01-11T21:05:01.2680373Z cca768f96df4: Waiting 2023-01-11T21:05:01.2680726Z 95c1ac011645: Pulling fs layer 2023-01-11T21:05:01.2681009Z 3046cc00c4ca: Pulling fs layer 2023-01-11T21:05:01.2681336Z 25571655d0e1: Waiting 2023-01-11T21:05:01.2681636Z 06369252d749: Waiting 2023-01-11T21:05:01.2681893Z 195d560d8cf6: Pulling fs layer 2023-01-11T21:05:01.2682254Z 77250abd5ca4: Pulling fs layer 2023-01-11T21:05:01.2682583Z 95c1ac011645: Waiting 2023-01-11T21:05:01.2682913Z 3046cc00c4ca: Waiting 2023-01-11T21:05:01.2683159Z 195d560d8cf6: Waiting 2023-01-11T21:05:01.2683513Z 881b24daf9c5: Pulling fs layer 2023-01-11T21:05:01.2683859Z 9fbf0a18619e: Pulling fs layer 2023-01-11T21:05:01.2684142Z 02048a597c22: Pulling fs layer 2023-01-11T21:05:01.2684470Z 859052a25d95: Pulling fs layer 2023-01-11T21:05:01.2684789Z 9fbf0a18619e: Waiting 2023-01-11T21:05:01.2685087Z 77250abd5ca4: Waiting 2023-01-11T21:05:01.2685397Z 19fde6a723a0: Waiting 2023-01-11T21:05:01.2685696Z 02048a597c22: Waiting 2023-01-11T21:05:01.2685991Z 3e03143da3c2: Pulling fs layer 2023-01-11T21:05:01.2686317Z 87d0ffa55850: Waiting 2023-01-11T21:05:01.2686655Z 61eecfa8b34e: Waiting 2023-01-11T21:05:01.4764739Z 3dc32ed140fb: Verifying Checksum 2023-01-11T21:05:01.4765500Z 3dc32ed140fb: Download complete 2023-01-11T21:05:01.5909784Z cf378b3cb3c7: Verifying Checksum 2023-01-11T21:05:01.5910606Z cf378b3cb3c7: Download complete 2023-01-11T21:05:01.6792284Z fb668870d8a7: Verifying Checksum 2023-01-11T21:05:01.6793060Z fb668870d8a7: Download complete 2023-01-11T21:05:01.8044197Z 9b4412378859: Verifying Checksum 2023-01-11T21:05:01.8045212Z 9b4412378859: Download complete 2023-01-11T21:05:01.9537802Z 54a1df240516: Verifying Checksum 2023-01-11T21:05:01.9538697Z 54a1df240516: Download complete 2023-01-11T21:05:02.0287346Z bdfd23ed3f48: Download complete 2023-01-11T21:05:02.1777913Z aee1dd761bdd: Download complete 2023-01-11T21:05:02.4673558Z fb668870d8a7: Pull complete 2023-01-11T21:05:02.4706705Z 5c7dd67e5809: Download complete 2023-01-11T21:05:02.7009211Z f8371ecb849a: Verifying Checksum 2023-01-11T21:05:02.7009779Z f8371ecb849a: Download complete 2023-01-11T21:05:02.7597270Z 3dc32ed140fb: Pull complete 2023-01-11T21:05:02.8187516Z ce4a87d45645: Download complete 2023-01-11T21:05:03.2765906Z 54a1df240516: Pull complete 2023-01-11T21:05:03.3773520Z cf378b3cb3c7: Pull complete 2023-01-11T21:05:03.4709540Z 9b4412378859: Pull complete 2023-01-11T21:05:10.5291465Z 39629f7269f9: Download complete 2023-01-11T21:05:10.6443588Z 87d0ffa55850: Download complete 2023-01-11T21:05:10.7532101Z 70702f8b5bc4: Verifying Checksum 2023-01-11T21:05:10.7532430Z 70702f8b5bc4: Download complete 2023-01-11T21:05:10.8558668Z 0c06be5c20e0: Verifying Checksum 2023-01-11T21:05:10.8558993Z 0c06be5c20e0: Download complete 2023-01-11T21:05:12.9309439Z 502253a1be21: Verifying Checksum 2023-01-11T21:05:12.9309787Z 502253a1be21: Download complete 2023-01-11T21:05:13.0846394Z 582d081a59fa: Verifying Checksum 2023-01-11T21:05:13.0846746Z 582d081a59fa: Download complete 2023-01-11T21:05:13.2256864Z e1c655e7ec0e: Verifying Checksum 2023-01-11T21:05:13.2257198Z e1c655e7ec0e: Download complete 2023-01-11T21:05:13.3799464Z b372c2a3bc3f: Verifying Checksum 2023-01-11T21:05:13.3799804Z b372c2a3bc3f: Download complete 2023-01-11T21:05:13.5011504Z 1c22f2f8c01b: Verifying Checksum 2023-01-11T21:05:13.5011851Z 1c22f2f8c01b: Download complete 2023-01-11T21:05:13.6052890Z b8f759fd0191: Verifying Checksum 2023-01-11T21:05:13.6053233Z b8f759fd0191: Download complete 2023-01-11T21:05:13.7251050Z e28e73a4bddd: Verifying Checksum 2023-01-11T21:05:13.7251374Z e28e73a4bddd: Download complete 2023-01-11T21:05:13.8413414Z 90d8f9bbe048: Verifying Checksum 2023-01-11T21:05:13.8413753Z 90d8f9bbe048: Download complete 2023-01-11T21:05:13.9257211Z b34bd39d0461: Verifying Checksum 2023-01-11T21:05:13.9257517Z b34bd39d0461: Download complete 2023-01-11T21:05:14.0180155Z 2f2308643d60: Verifying Checksum 2023-01-11T21:05:14.0180430Z 2f2308643d60: Download complete 2023-01-11T21:05:16.5573715Z 5feda9af2542: Verifying Checksum 2023-01-11T21:05:16.5574331Z 5feda9af2542: Download complete 2023-01-11T21:05:16.6447386Z 9ea746919509: Verifying Checksum 2023-01-11T21:05:16.6447693Z 9ea746919509: Download complete 2023-01-11T21:05:16.7448855Z 1a2fd7b216d7: Verifying Checksum 2023-01-11T21:05:16.7449159Z 1a2fd7b216d7: Download complete 2023-01-11T21:05:16.9049567Z 19fde6a723a0: Verifying Checksum 2023-01-11T21:05:16.9049875Z 19fde6a723a0: Download complete 2023-01-11T21:05:16.9777586Z 06369252d749: Verifying Checksum 2023-01-11T21:05:16.9777872Z 06369252d749: Download complete 2023-01-11T21:05:17.0888205Z ea4bfeaa0fc7: Verifying Checksum 2023-01-11T21:05:17.0888506Z ea4bfeaa0fc7: Download complete 2023-01-11T21:05:17.4619333Z 8e3432e5a569: Verifying Checksum 2023-01-11T21:05:17.4619683Z 8e3432e5a569: Download complete 2023-01-11T21:05:17.5697133Z f550b7ff2470: Verifying Checksum 2023-01-11T21:05:17.5697721Z f550b7ff2470: Download complete 2023-01-11T21:05:17.7080951Z 12ddc57b99eb: Verifying Checksum 2023-01-11T21:05:17.7081283Z 12ddc57b99eb: Download complete 2023-01-11T21:05:19.2073573Z 8345085fb0a0: Verifying Checksum 2023-01-11T21:05:19.2074010Z 8345085fb0a0: Download complete 2023-01-11T21:05:19.3271131Z 4cc94dbec031: Download complete 2023-01-11T21:05:19.4396899Z 29a7c0d5fa4c: Verifying Checksum 2023-01-11T21:05:19.4397252Z 29a7c0d5fa4c: Download complete 2023-01-11T21:05:20.3564394Z 25571655d0e1: Verifying Checksum 2023-01-11T21:05:20.3564783Z 25571655d0e1: Download complete 2023-01-11T21:05:20.4550174Z bdf297d7f88c: Verifying Checksum 2023-01-11T21:05:20.4550673Z bdf297d7f88c: Download complete 2023-01-11T21:05:21.9544271Z 0b3950af8ae1: Verifying Checksum 2023-01-11T21:05:21.9544749Z 0b3950af8ae1: Download complete 2023-01-11T21:05:22.0618143Z 6d68f7da8baa: Verifying Checksum 2023-01-11T21:05:22.0618456Z 6d68f7da8baa: Download complete 2023-01-11T21:05:22.1449121Z cca768f96df4: Verifying Checksum 2023-01-11T21:05:22.1449442Z cca768f96df4: Download complete 2023-01-11T21:05:24.8682494Z a1d16b6a5070: Verifying Checksum 2023-01-11T21:05:24.8682877Z a1d16b6a5070: Download complete 2023-01-11T21:05:24.9532839Z 61eecfa8b34e: Verifying Checksum 2023-01-11T21:05:24.9533163Z 61eecfa8b34e: Download complete 2023-01-11T21:05:25.0308455Z 95c1ac011645: Verifying Checksum 2023-01-11T21:05:25.0308950Z 95c1ac011645: Download complete 2023-01-11T21:05:25.1302934Z 3046cc00c4ca: Download complete 2023-01-11T21:05:25.2170226Z 195d560d8cf6: Verifying Checksum 2023-01-11T21:05:25.2171119Z 195d560d8cf6: Download complete 2023-01-11T21:05:25.9773540Z 77250abd5ca4: Verifying Checksum 2023-01-11T21:05:25.9773875Z 77250abd5ca4: Download complete 2023-01-11T21:05:26.0578380Z 881b24daf9c5: Verifying Checksum 2023-01-11T21:05:26.0578951Z 881b24daf9c5: Download complete 2023-01-11T21:05:26.7677329Z 502253a1be21: Pull complete 2023-01-11T21:05:26.8699680Z 5c7dd67e5809: Pull complete 2023-01-11T21:05:26.9860429Z bdfd23ed3f48: Pull complete 2023-01-11T21:05:27.1186378Z aee1dd761bdd: Pull complete 2023-01-11T21:05:27.8764195Z 9fbf0a18619e: Verifying Checksum 2023-01-11T21:05:27.8764588Z 9fbf0a18619e: Download complete 2023-01-11T21:05:27.9558191Z 02048a597c22: Verifying Checksum 2023-01-11T21:05:31.4457859Z 8c3cf3d5e1c5: Verifying Checksum 2023-01-11T21:05:31.4458259Z 8c3cf3d5e1c5: Download complete 2023-01-11T21:05:31.5243352Z 3e03143da3c2: Verifying Checksum 2023-01-11T21:05:31.5243673Z 3e03143da3c2: Download complete 2023-01-11T21:05:49.3045408Z 5feda9af2542: Pull complete 2023-01-11T21:05:51.2432814Z f8371ecb849a: Pull complete 2023-01-11T21:05:53.1512033Z ce4a87d45645: Pull complete 2023-01-11T21:06:00.4722772Z 39629f7269f9: Pull complete 2023-01-11T21:06:02.3200108Z 87d0ffa55850: Pull complete 2023-01-11T21:06:04.2236307Z 70702f8b5bc4: Pull complete 2023-01-11T21:06:06.7072434Z 0c06be5c20e0: Pull complete 2023-01-11T21:06:10.6332556Z b372c2a3bc3f: Pull complete 2023-01-11T21:06:12.6119044Z c7726d39d806: Verifying Checksum 2023-01-11T21:06:12.6119381Z c7726d39d806: Download complete 2023-01-11T21:06:13.0327371Z 582d081a59fa: Pull complete 2023-01-11T21:06:15.0078517Z e1c655e7ec0e: Pull complete 2023-01-11T21:06:47.4227343Z c7726d39d806: Pull complete 2023-01-11T21:06:49.2966707Z 1c22f2f8c01b: Pull complete 2023-01-11T21:06:51.2008962Z b8f759fd0191: Pull complete 2023-01-11T21:06:53.2322720Z e28e73a4bddd: Pull complete 2023-01-11T21:06:55.7152320Z 90d8f9bbe048: Pull complete 2023-01-11T21:06:57.4500905Z b34bd39d0461: Pull complete 2023-01-11T21:06:59.3081719Z 2f2308643d60: Pull complete 2023-01-11T21:07:01.7918969Z 859052a25d95: Verifying Checksum 2023-01-11T21:07:01.7919298Z 859052a25d95: Download complete 2023-01-11T21:07:03.3651007Z 8e3432e5a569: Pull complete 2023-01-11T21:07:05.4759036Z 9ea746919509: Pull complete 2023-01-11T21:07:07.0705986Z 1a2fd7b216d7: Pull complete 2023-01-11T21:07:10.1637011Z 19fde6a723a0: Pull complete 2023-01-11T21:07:13.2812410Z 06369252d749: Pull complete 2023-01-11T21:07:16.2326800Z ea4bfeaa0fc7: Pull complete 2023-01-11T21:07:24.7926379Z a1d16b6a5070: Pull complete 2023-01-11T21:07:26.7346816Z f550b7ff2470: Pull complete 2023-01-11T21:07:28.6738399Z 12ddc57b99eb: Pull complete 2023-01-11T21:07:31.5254470Z 8345085fb0a0: Pull complete 2023-01-11T21:07:32.0020199Z 4cc94dbec031: Pull complete 2023-01-11T21:07:32.1104184Z 29a7c0d5fa4c: Pull complete 2023-01-11T21:07:32.4709477Z 25571655d0e1: Pull complete 2023-01-11T21:07:32.5755973Z bdf297d7f88c: Pull complete 2023-01-11T21:07:33.9682243Z 0b3950af8ae1: Pull complete 2023-01-11T21:07:34.0673448Z 6d68f7da8baa: Pull complete 2023-01-11T21:07:34.1809771Z cca768f96df4: Pull complete 2023-01-11T21:07:40.5706079Z 8c3cf3d5e1c5: Pull complete 2023-01-11T21:07:42.4730611Z 61eecfa8b34e: Pull complete 2023-01-11T21:07:43.9988713Z 95c1ac011645: Pull complete 2023-01-11T21:07:44.0963565Z 3046cc00c4ca: Pull complete 2023-01-11T21:07:44.1949789Z 195d560d8cf6: Pull complete 2023-01-11T21:07:45.0045302Z 77250abd5ca4: Pull complete 2023-01-11T21:07:45.1073143Z 881b24daf9c5: Pull complete 2023-01-11T21:07:47.0189510Z 9fbf0a18619e: Pull complete 2023-01-11T21:07:47.1202202Z 02048a597c22: Pull complete 2023-01-11T21:08:29.0821185Z 859052a25d95: Pull complete 2023-01-11T21:08:30.9928076Z 3e03143da3c2: Pull complete 2023-01-11T21:08:32.2787230Z Digest: sha256:866df6c1171dbe014496717cf2080d6cc72ca611a4e8146525c9ef09640c8ba4 2023-01-11T21:08:32.7783797Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd 2023-01-11T21:08:32.9774903Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd 2023-01-11T21:08:33.0589721Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2023-01-11T21:08:33.0590081Z with: 2023-01-11T21:08:33.0590323Z driver-version: 515.76 2023-01-11T21:08:33.0590545Z env: 2023-01-11T21:08:33.0590785Z GIT_DEFAULT_BRANCH: master 2023-01-11T21:08:33.0591039Z ##[endgroup] 2023-01-11T21:08:33.0993937Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2023-01-11T21:08:33.0994252Z with: 2023-01-11T21:08:33.0994485Z timeout_minutes: 10 2023-01-11T21:08:33.0994717Z max_attempts: 3 2023-01-11T21:08:33.1001444Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all" >> "${GITHUB_ENV}" 2023-01-11T21:08:33.1008402Z retry_wait_seconds: 10 2023-01-11T21:08:33.1008668Z polling_interval_seconds: 1 2023-01-11T21:08:33.1008999Z warning_on_retry: true 2023-01-11T21:08:33.1009270Z continue_on_error: false 2023-01-11T21:08:33.1009497Z env: 2023-01-11T21:08:33.1009734Z GIT_DEFAULT_BRANCH: master 2023-01-11T21:08:33.1010003Z DRIVER_VERSION: 515.76 2023-01-11T21:08:33.1010235Z ##[endgroup] 2023-01-11T21:08:33.1681680Z == Installing nvidia driver NVIDIA-Linux-x86_64-515.76.run == 2023-01-11T21:08:33.1683728Z + pre_install_nvidia_driver_amzn2 2023-01-11T21:08:33.1684137Z + sudo yum remove -y nvidia-driver-latest-dkms 2023-01-11T21:08:33.6383186Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2023-01-11T21:08:33.6810326Z No Match for argument: nvidia-driver-latest-dkms 2023-01-11T21:08:33.7099466Z No Packages marked for removal 2023-01-11T21:08:33.7257083Z + install_nvidia_driver_common 2023-01-11T21:08:33.7260829Z + echo 'Before installing NVIDIA driver' 2023-01-11T21:08:33.7261473Z Before installing NVIDIA driver 2023-01-11T21:08:33.7263665Z + lspci 2023-01-11T21:08:34.8970030Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 2023-01-11T21:08:34.8970904Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2023-01-11T21:08:34.8971821Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 2023-01-11T21:08:34.8972250Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01) 2023-01-11T21:08:34.8972618Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 2023-01-11T21:08:34.8975706Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2023-01-11T21:08:34.8976142Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2023-01-11T21:08:34.8976579Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2023-01-11T21:08:34.8976996Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01) 2023-01-11T21:08:34.8977294Z + lsmod 2023-01-11T21:08:34.8994766Z Module Size Used by 2023-01-11T21:08:34.8995070Z xt_conntrack 16384 1 2023-01-11T21:08:34.8995345Z ipt_MASQUERADE 16384 1 2023-01-11T21:08:34.8995628Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2023-01-11T21:08:34.8995943Z nf_conntrack_netlink 49152 0 2023-01-11T21:08:34.8996244Z nfnetlink 16384 2 nf_conntrack_netlink 2023-01-11T21:08:34.8996507Z xfrm_user 45056 1 2023-01-11T21:08:34.8996773Z xfrm_algo 16384 1 xfrm_user 2023-01-11T21:08:34.8997041Z xt_addrtype 16384 2 2023-01-11T21:08:34.8997288Z iptable_filter 16384 1 2023-01-11T21:08:34.8997581Z iptable_nat 16384 1 2023-01-11T21:08:34.8997850Z nf_conntrack_ipv4 16384 3 2023-01-11T21:08:34.8998124Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2023-01-11T21:08:34.8998434Z nf_nat_ipv4 16384 1 iptable_nat 2023-01-11T21:08:34.8998751Z nf_nat 36864 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2023-01-11T21:08:34.8999222Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2023-01-11T21:08:34.8999612Z br_netfilter 24576 0 2023-01-11T21:08:34.8999889Z bridge 172032 1 br_netfilter 2023-01-11T21:08:34.9000157Z stp 16384 1 bridge 2023-01-11T21:08:34.9000784Z llc 16384 2 bridge,stp 2023-01-11T21:08:34.9001060Z overlay 86016 0 2023-01-11T21:08:34.9001326Z sunrpc 393216 1 2023-01-11T21:08:34.9001592Z dm_mirror 28672 0 2023-01-11T21:08:34.9001840Z dm_region_hash 20480 1 dm_mirror 2023-01-11T21:08:34.9002138Z dm_log 20480 2 dm_region_hash,dm_mirror 2023-01-11T21:08:34.9002444Z dm_mod 143360 2 dm_log,dm_mirror 2023-01-11T21:08:34.9002692Z dax 69632 1 dm_mod 2023-01-11T21:08:34.9002945Z sb_edac 24576 0 2023-01-11T21:08:34.9003202Z crc32_pclmul 16384 0 2023-01-11T21:08:34.9003456Z ghash_clmulni_intel 16384 0 2023-01-11T21:08:34.9003909Z pcbc 16384 0 2023-01-11T21:08:34.9004162Z aesni_intel 188416 0 2023-01-11T21:08:34.9004414Z aes_x86_64 20480 1 aesni_intel 2023-01-11T21:08:34.9004677Z ata_piix 36864 0 2023-01-11T21:08:34.9004954Z crypto_simd 16384 1 aesni_intel 2023-01-11T21:08:34.9005224Z glue_helper 16384 1 aesni_intel 2023-01-11T21:08:34.9005498Z pcc_cpufreq 16384 0 2023-01-11T21:08:34.9005764Z libata 266240 1 ata_piix 2023-01-11T21:08:34.9006007Z mousedev 24576 0 2023-01-11T21:08:34.9006550Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2023-01-11T21:08:34.9006871Z evdev 20480 3 2023-01-11T21:08:34.9007126Z scsi_mod 245760 1 libata 2023-01-11T21:08:34.9007379Z psmouse 32768 0 2023-01-11T21:08:34.9007629Z button 16384 0 2023-01-11T21:08:34.9007982Z ena 114688 0 2023-01-11T21:08:34.9008237Z xen_blkfront 49152 2 2023-01-11T21:08:34.9008488Z crc32c_intel 24576 0 2023-01-11T21:08:34.9008737Z autofs4 49152 2 2023-01-11T21:08:34.9008965Z + modinfo nvidia 2023-01-11T21:08:34.9009242Z modinfo: ERROR: Module nvidia not found. 2023-01-11T21:08:34.9009509Z + true 2023-01-11T21:08:34.9009722Z + HAS_NVIDIA_DRIVER=0 2023-01-11T21:08:34.9012345Z ++ command -v nvidia-smi 2023-01-11T21:08:34.9014221Z + '[' -x '' ']' 2023-01-11T21:08:34.9015257Z + '[' 0 -eq 0 ']' 2023-01-11T21:08:34.9015859Z + '[' amzn2 '!=' ubuntu20.04 ']' 2023-01-11T21:08:34.9016577Z + sudo yum groupinstall -y 'Development Tools' 2023-01-11T21:08:35.3680940Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2023-01-11T21:08:35.6898053Z Resolving Dependencies 2023-01-11T21:08:35.6903220Z --> Running transaction check 2023-01-11T21:08:35.6906149Z ---> Package autoconf.noarch 0:2.69-11.amzn2 will be installed 2023-01-11T21:08:35.7115489Z --> Processing Dependency: m4 >= 1.4.14 for package: autoconf-2.69-11.amzn2.noarch 2023-01-11T21:08:35.9252603Z --> Processing Dependency: perl(Data::Dumper) for package: autoconf-2.69-11.amzn2.noarch 2023-01-11T21:08:35.9254892Z ---> Package automake.noarch 0:1.13.4-3.1.amzn2 will be installed 2023-01-11T21:08:35.9300423Z --> Processing Dependency: perl(Thread::Queue) for package: automake-1.13.4-3.1.amzn2.noarch 2023-01-11T21:08:35.9307468Z --> Processing Dependency: perl(TAP::Parser) for package: automake-1.13.4-3.1.amzn2.noarch 2023-01-11T21:08:35.9318232Z ---> Package bison.x86_64 0:3.0.4-6.amzn2.0.2 will be installed 2023-01-11T21:08:35.9431998Z ---> Package byacc.x86_64 0:1.9.20130304-3.amzn2.0.2 will be installed 2023-01-11T21:08:35.9439145Z ---> Package cscope.x86_64 0:15.8-10.amzn2.0.2 will be installed 2023-01-11T21:08:35.9484089Z --> Processing Dependency: emacs-filesystem for package: cscope-15.8-10.amzn2.0.2.x86_64 2023-01-11T21:08:35.9508389Z ---> Package ctags.x86_64 0:5.8-13.amzn2.0.2 will be installed 2023-01-11T21:08:35.9517270Z ---> Package diffstat.x86_64 0:1.57-4.amzn2.0.2 will be installed 2023-01-11T21:08:35.9525186Z ---> Package doxygen.x86_64 1:1.8.5-4.amzn2 will be installed 2023-01-11T21:08:35.9625888Z ---> Package elfutils.x86_64 0:0.176-2.amzn2 will be installed 2023-01-11T21:08:35.9762397Z ---> Package flex.x86_64 0:2.5.37-3.amzn2.0.3 will be installed 2023-01-11T21:08:35.9782290Z ---> Package gcc.x86_64 0:7.3.1-15.amzn2 will be installed 2023-01-11T21:08:35.9954380Z --> Processing Dependency: cpp = 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64 2023-01-11T21:08:35.9973970Z --> Processing Dependency: libsanitizer >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64 2023-01-11T21:08:36.0028747Z --> Processing Dependency: libquadmath >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64 2023-01-11T21:08:36.0080523Z --> Processing Dependency: libmpx >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64 2023-01-11T21:08:36.0135409Z --> Processing Dependency: libitm >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64 2023-01-11T21:08:36.0188262Z --> Processing Dependency: libcilkrts >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64 2023-01-11T21:08:36.0241364Z --> Processing Dependency: libatomic >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64 2023-01-11T21:08:36.0295812Z --> Processing Dependency: glibc-devel >= 2.2.90-12 for package: gcc-7.3.1-15.amzn2.x86_64 2023-01-11T21:08:36.0459522Z --> Processing Dependency: libmpfr.so.4()(64bit) for package: gcc-7.3.1-15.amzn2.x86_64 2023-01-11T21:08:36.0480230Z --> Processing Dependency: libmpc.so.3()(64bit) for package: gcc-7.3.1-15.amzn2.x86_64 2023-01-11T21:08:36.0501957Z ---> Package gcc-c++.x86_64 0:7.3.1-15.amzn2 will be installed 2023-01-11T21:08:36.0529049Z ---> Package gcc-gfortran.x86_64 0:7.3.1-15.amzn2 will be installed 2023-01-11T21:08:36.0563521Z --> Processing Dependency: libgfortran.so.4()(64bit) for package: gcc-gfortran-7.3.1-15.amzn2.x86_64 2023-01-11T21:08:36.0633382Z ---> Package indent.x86_64 0:2.2.11-13.amzn2.0.2 will be installed 2023-01-11T21:08:36.0647590Z ---> Package intltool.noarch 0:0.50.2-7.amzn2 will be installed 2023-01-11T21:08:36.0694329Z --> Processing Dependency: perl(XML::Parser) for package: intltool-0.50.2-7.amzn2.noarch 2023-01-11T21:08:36.0709397Z --> Processing Dependency: gettext-devel for package: intltool-0.50.2-7.amzn2.noarch 2023-01-11T21:08:36.0727883Z ---> Package libtool.x86_64 0:2.4.2-22.2.amzn2.0.2 will be installed 2023-01-11T21:08:36.0757345Z ---> Package patch.x86_64 0:2.7.1-12.amzn2.0.2 will be installed 2023-01-11T21:08:36.0801091Z ---> Package patchutils.x86_64 0:0.3.3-4.amzn2.0.1 will be installed 2023-01-11T21:08:36.0824151Z ---> Package rcs.x86_64 0:5.9.0-5.amzn2.0.2 will be installed 2023-01-11T21:08:36.0856551Z ---> Package rpm-build.x86_64 0:4.11.3-48.amzn2.0.2 will be installed 2023-01-11T21:08:36.1087654Z --> Processing Dependency: /usr/bin/gdb-add-index for package: rpm-build-4.11.3-48.amzn2.0.2.x86_64 2023-01-11T21:08:36.1105543Z ---> Package rpm-sign.x86_64 0:4.11.3-48.amzn2.0.2 will be installed 2023-01-11T21:08:36.1128855Z ---> Package subversion.x86_64 0:1.7.14-16.amzn2.0.1 will be installed 2023-01-11T21:08:36.1302413Z --> Processing Dependency: subversion-libs(x86-64) = 1.7.14-16.amzn2.0.1 for package: subversion-1.7.14-16.amzn2.0.1.x86_64 2023-01-11T21:08:36.1322541Z --> Processing Dependency: libsvn_wc-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64 2023-01-11T21:08:36.1324207Z --> Processing Dependency: libsvn_subr-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64 2023-01-11T21:08:36.1325468Z --> Processing Dependency: libsvn_repos-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64 2023-01-11T21:08:36.1326673Z --> Processing Dependency: libsvn_ra_svn-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64 2023-01-11T21:08:36.1327928Z --> Processing Dependency: libsvn_ra_neon-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64 2023-01-11T21:08:36.1329171Z --> Processing Dependency: libsvn_ra_local-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64 2023-01-11T21:08:36.1330670Z --> Processing Dependency: libsvn_ra-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64 2023-01-11T21:08:36.1331903Z --> Processing Dependency: libsvn_fs_util-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64 2023-01-11T21:08:36.1332645Z --> Processing Dependency: libsvn_fs_fs-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64 2023-01-11T21:08:36.1333272Z --> Processing Dependency: libsvn_fs_base-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64 2023-01-11T21:08:36.1333889Z --> Processing Dependency: libsvn_fs-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64 2023-01-11T21:08:36.1334479Z --> Processing Dependency: libsvn_diff-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64 2023-01-11T21:08:36.1335102Z --> Processing Dependency: libsvn_delta-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64 2023-01-11T21:08:36.1335881Z --> Processing Dependency: libsvn_client-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64 2023-01-11T21:08:36.1336459Z --> Processing Dependency: libneon.so.27()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64 2023-01-11T21:08:36.1351242Z --> Processing Dependency: libaprutil-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64 2023-01-11T21:08:36.1371513Z --> Processing Dependency: libapr-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64 2023-01-11T21:08:36.1395706Z ---> Package swig.x86_64 0:3.0.12-11.amzn2.0.3 will be installed 2023-01-11T21:08:36.1416935Z ---> Package system-rpm-config.noarch 0:9.1.0-76.amzn2.0.14 will be installed 2023-01-11T21:08:36.1463783Z --> Processing Dependency: dwz >= 0.4 for package: system-rpm-config-9.1.0-76.amzn2.0.14.noarch 2023-01-11T21:08:36.1480705Z --> Processing Dependency: perl-srpm-macros for package: system-rpm-config-9.1.0-76.amzn2.0.14.noarch 2023-01-11T21:08:36.1493297Z --> Processing Dependency: go-srpm-macros for package: system-rpm-config-9.1.0-76.amzn2.0.14.noarch 2023-01-11T21:08:36.1672257Z ---> Package systemtap.x86_64 0:4.5-1.amzn2.0.1 will be installed 2023-01-11T21:08:36.1685242Z --> Processing Dependency: systemtap-devel = 4.5-1.amzn2.0.1 for package: systemtap-4.5-1.amzn2.0.1.x86_64 2023-01-11T21:08:36.1706661Z --> Processing Dependency: systemtap-client = 4.5-1.amzn2.0.1 for package: systemtap-4.5-1.amzn2.0.1.x86_64 2023-01-11T21:08:36.1720247Z --> Running transaction check 2023-01-11T21:08:36.1723537Z ---> Package apr.x86_64 0:1.7.0-9.amzn2 will be installed 2023-01-11T21:08:36.1799726Z ---> Package apr-util.x86_64 0:1.6.1-5.amzn2.0.2 will be installed 2023-01-11T21:08:36.1838210Z --> Processing Dependency: apr-util-bdb(x86-64) = 1.6.1-5.amzn2.0.2 for package: apr-util-1.6.1-5.amzn2.0.2.x86_64 2023-01-11T21:08:36.1853169Z ---> Package cpp.x86_64 0:7.3.1-15.amzn2 will be installed 2023-01-11T21:08:36.1926222Z ---> Package dwz.x86_64 0:0.11-3.amzn2.0.3 will be installed 2023-01-11T21:08:36.1937433Z ---> Package emacs-filesystem.noarch 1:27.2-4.amzn2.0.1 will be installed 2023-01-11T21:08:36.1938561Z ---> Package gdb.x86_64 0:8.0.1-36.amzn2.0.1 will be installed 2023-01-11T21:08:36.2008743Z ---> Package gettext-devel.x86_64 0:0.19.8.1-3.amzn2 will be installed 2023-01-11T21:08:36.2066977Z --> Processing Dependency: gettext-common-devel = 0.19.8.1-3.amzn2 for package: gettext-devel-0.19.8.1-3.amzn2.x86_64 2023-01-11T21:08:36.2075861Z ---> Package glibc-devel.x86_64 0:2.26-62.amzn2 will be installed 2023-01-11T21:08:36.2196643Z --> Processing Dependency: glibc-headers = 2.26-62.amzn2 for package: glibc-devel-2.26-62.amzn2.x86_64 2023-01-11T21:08:36.2225044Z --> Processing Dependency: glibc-headers for package: glibc-devel-2.26-62.amzn2.x86_64 2023-01-11T21:08:36.2225627Z ---> Package go-srpm-macros.noarch 0:3.0.15-23.amzn2.0.2 will be installed 2023-01-11T21:08:36.2230305Z ---> Package libatomic.x86_64 0:7.3.1-15.amzn2 will be installed 2023-01-11T21:08:36.2243708Z ---> Package libcilkrts.x86_64 0:7.3.1-15.amzn2 will be installed 2023-01-11T21:08:36.2270512Z ---> Package libgfortran.x86_64 0:7.3.1-15.amzn2 will be installed 2023-01-11T21:08:36.2306290Z ---> Package libitm.x86_64 0:7.3.1-15.amzn2 will be installed 2023-01-11T21:08:36.2322200Z ---> Package libmpc.x86_64 0:1.0.1-3.amzn2.0.2 will be installed 2023-01-11T21:08:36.2334958Z ---> Package libmpx.x86_64 0:7.3.1-15.amzn2 will be installed 2023-01-11T21:08:36.2349769Z ---> Package libquadmath.x86_64 0:7.3.1-15.amzn2 will be installed 2023-01-11T21:08:36.2375298Z ---> Package libsanitizer.x86_64 0:7.3.1-15.amzn2 will be installed 2023-01-11T21:08:36.2420725Z ---> Package m4.x86_64 0:1.4.16-10.amzn2.0.2 will be installed 2023-01-11T21:08:36.2435726Z ---> Package mpfr.x86_64 0:3.1.1-4.amzn2.0.2 will be installed 2023-01-11T21:08:36.2457093Z ---> Package neon.x86_64 0:0.30.0-3.amzn2.0.2 will be installed 2023-01-11T21:08:36.2531841Z --> Processing Dependency: libgnutls.so.28(GNUTLS_2_12)(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64 2023-01-11T21:08:36.2570725Z --> Processing Dependency: libgnutls.so.28(GNUTLS_1_4)(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64 2023-01-11T21:08:36.2571572Z --> Processing Dependency: libproxy.so.1()(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64 2023-01-11T21:08:36.2590920Z --> Processing Dependency: libpakchois.so.0()(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64 2023-01-11T21:08:36.2608368Z --> Processing Dependency: libgnutls.so.28()(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64 2023-01-11T21:08:36.2615016Z ---> Package perl-Data-Dumper.x86_64 0:2.145-3.amzn2.0.2 will be installed 2023-01-11T21:08:36.2662134Z ---> Package perl-Test-Harness.noarch 0:3.28-3.amzn2 will be installed 2023-01-11T21:08:36.2756566Z ---> Package perl-Thread-Queue.noarch 0:3.02-2.amzn2 will be installed 2023-01-11T21:08:36.2769218Z ---> Package perl-XML-Parser.x86_64 0:2.41-10.amzn2.0.2 will be installed 2023-01-11T21:08:36.2785004Z ---> Package perl-srpm-macros.noarch 0:1-8.amzn2.0.1 will be installed 2023-01-11T21:08:36.2786066Z ---> Package subversion-libs.x86_64 0:1.7.14-16.amzn2.0.1 will be installed 2023-01-11T21:08:36.2814556Z ---> Package systemtap-client.x86_64 0:4.5-1.amzn2.0.1 will be installed 2023-01-11T21:08:36.3025060Z --> Processing Dependency: mokutil for package: systemtap-client-4.5-1.amzn2.0.1.x86_64 2023-01-11T21:08:36.3038449Z --> Processing Dependency: libavahi-common.so.3()(64bit) for package: systemtap-client-4.5-1.amzn2.0.1.x86_64 2023-01-11T21:08:36.3063980Z --> Processing Dependency: libavahi-client.so.3()(64bit) for package: systemtap-client-4.5-1.amzn2.0.1.x86_64 2023-01-11T21:08:36.3065065Z ---> Package systemtap-devel.x86_64 0:4.5-1.amzn2.0.1 will be installed 2023-01-11T21:08:36.3183637Z --> Processing Dependency: kernel-devel-uname-r for package: systemtap-devel-4.5-1.amzn2.0.1.x86_64 2023-01-11T21:08:36.4259672Z --> Running transaction check 2023-01-11T21:08:36.4260713Z ---> Package apr-util-bdb.x86_64 0:1.6.1-5.amzn2.0.2 will be installed 2023-01-11T21:08:36.4270315Z ---> Package avahi-libs.x86_64 0:0.6.31-20.amzn2 will be installed 2023-01-11T21:08:36.4297011Z ---> Package gettext-common-devel.noarch 0:0.19.8.1-3.amzn2 will be installed 2023-01-11T21:08:36.4298154Z ---> Package glibc-headers.x86_64 0:2.26-62.amzn2 will be installed 2023-01-11T21:08:36.4372939Z --> Processing Dependency: kernel-headers >= 2.2.1 for package: glibc-headers-2.26-62.amzn2.x86_64 2023-01-11T21:08:36.5525474Z --> Processing Dependency: kernel-headers for package: glibc-headers-2.26-62.amzn2.x86_64 2023-01-11T21:08:36.5526215Z ---> Package gnutls.x86_64 0:3.3.29-9.amzn2.0.1 will be installed 2023-01-11T21:08:36.5592236Z --> Processing Dependency: trousers >= 0.3.11.2 for package: gnutls-3.3.29-9.amzn2.0.1.x86_64 2023-01-11T21:08:36.5617472Z ---> Package kernel-devel.x86_64 0:4.14.301-224.520.amzn2 will be installed 2023-01-11T21:08:36.5644919Z --> Processing Dependency: elfutils-libelf-devel for package: kernel-devel-4.14.301-224.520.amzn2.x86_64 2023-01-11T21:08:36.5664796Z ---> Package libproxy.x86_64 0:0.4.11-10.amzn2.0.3 will be installed 2023-01-11T21:08:36.5693195Z --> Processing Dependency: libmodman.so.1()(64bit) for package: libproxy-0.4.11-10.amzn2.0.3.x86_64 2023-01-11T21:08:36.5710943Z ---> Package mokutil.x86_64 1:0.3.0-10.amzn2.0.1 will be installed 2023-01-11T21:08:36.5758993Z --> Processing Dependency: libefivar.so.1(libefivar.so.0)(64bit) for package: 1:mokutil-0.3.0-10.amzn2.0.1.x86_64 2023-01-11T21:08:36.5779510Z --> Processing Dependency: libefivar.so.1(LIBEFIVAR_0.24)(64bit) for package: 1:mokutil-0.3.0-10.amzn2.0.1.x86_64 2023-01-11T21:08:36.5780707Z --> Processing Dependency: libefivar.so.1()(64bit) for package: 1:mokutil-0.3.0-10.amzn2.0.1.x86_64 2023-01-11T21:08:36.5781360Z ---> Package pakchois.x86_64 0:0.4-10.amzn2.0.2 will be installed 2023-01-11T21:08:36.5794808Z --> Running transaction check 2023-01-11T21:08:36.5796068Z ---> Package efivar-libs.x86_64 0:31-4.amzn2.0.4 will be installed 2023-01-11T21:08:36.5813436Z ---> Package elfutils-libelf-devel.x86_64 0:0.176-2.amzn2 will be installed 2023-01-11T21:08:36.5825983Z --> Processing Dependency: pkgconfig(zlib) for package: elfutils-libelf-devel-0.176-2.amzn2.x86_64 2023-01-11T21:08:36.5852915Z ---> Package kernel-headers.x86_64 0:4.14.301-224.520.amzn2 will be installed 2023-01-11T21:08:36.5854093Z ---> Package libmodman.x86_64 0:2.0.1-8.amzn2.0.2 will be installed 2023-01-11T21:08:36.5880396Z ---> Package trousers.x86_64 0:0.3.14-2.amzn2.0.2 will be installed 2023-01-11T21:08:36.5938383Z --> Running transaction check 2023-01-11T21:08:36.5938812Z ---> Package zlib-devel.x86_64 0:1.2.7-19.amzn2.0.2 will be installed 2023-01-11T21:08:36.8532090Z --> Finished Dependency Resolution 2023-01-11T21:08:36.9279561Z 2023-01-11T21:08:36.9280252Z Dependencies Resolved 2023-01-11T21:08:36.9394323Z 2023-01-11T21:08:36.9394795Z ================================================================================ 2023-01-11T21:08:36.9395164Z Package Arch Version Repository Size 2023-01-11T21:08:36.9395712Z ================================================================================ 2023-01-11T21:08:36.9396093Z Installing for group install "Development Tools": 2023-01-11T21:08:36.9396640Z autoconf noarch 2.69-11.amzn2 amzn2-core 701 k 2023-01-11T21:08:36.9397071Z automake noarch 1.13.4-3.1.amzn2 amzn2-core 679 k 2023-01-11T21:08:36.9397501Z bison x86_64 3.0.4-6.amzn2.0.2 amzn2-core 674 k 2023-01-11T21:08:36.9397939Z byacc x86_64 1.9.20130304-3.amzn2.0.2 amzn2-core 66 k 2023-01-11T21:08:36.9398346Z cscope x86_64 15.8-10.amzn2.0.2 amzn2-core 204 k 2023-01-11T21:08:36.9398777Z ctags x86_64 5.8-13.amzn2.0.2 amzn2-core 157 k 2023-01-11T21:08:36.9399210Z diffstat x86_64 1.57-4.amzn2.0.2 amzn2-core 35 k 2023-01-11T21:08:36.9399650Z doxygen x86_64 1:1.8.5-4.amzn2 amzn2-core 3.5 M 2023-01-11T21:08:36.9400058Z elfutils x86_64 0.176-2.amzn2 amzn2-core 307 k 2023-01-11T21:08:36.9400477Z flex x86_64 2.5.37-3.amzn2.0.3 amzn2-core 291 k 2023-01-11T21:08:36.9400894Z gcc x86_64 7.3.1-15.amzn2 amzn2-core 22 M 2023-01-11T21:08:36.9401296Z gcc-c++ x86_64 7.3.1-15.amzn2 amzn2-core 13 M 2023-01-11T21:08:36.9401722Z gcc-gfortran x86_64 7.3.1-15.amzn2 amzn2-core 11 M 2023-01-11T21:08:36.9402155Z indent x86_64 2.2.11-13.amzn2.0.2 amzn2-core 150 k 2023-01-11T21:08:36.9402589Z intltool noarch 0.50.2-7.amzn2 amzn2-core 59 k 2023-01-11T21:08:36.9402997Z libtool x86_64 2.4.2-22.2.amzn2.0.2 amzn2-core 588 k 2023-01-11T21:08:36.9403423Z patch x86_64 2.7.1-12.amzn2.0.2 amzn2-core 110 k 2023-01-11T21:08:36.9403862Z patchutils x86_64 0.3.3-4.amzn2.0.1 amzn2-core 104 k 2023-01-11T21:08:36.9404284Z rcs x86_64 5.9.0-5.amzn2.0.2 amzn2-core 231 k 2023-01-11T21:08:36.9404708Z rpm-build x86_64 4.11.3-48.amzn2.0.2 amzn2-core 150 k 2023-01-11T21:08:36.9405139Z rpm-sign x86_64 4.11.3-48.amzn2.0.2 amzn2-core 50 k 2023-01-11T21:08:36.9405568Z subversion x86_64 1.7.14-16.amzn2.0.1 amzn2-core 1.0 M 2023-01-11T21:08:36.9405976Z swig x86_64 3.0.12-11.amzn2.0.3 amzn2-core 1.4 M 2023-01-11T21:08:36.9406412Z system-rpm-config noarch 9.1.0-76.amzn2.0.14 amzn2-core 90 k 2023-01-11T21:08:36.9406861Z systemtap x86_64 4.5-1.amzn2.0.1 amzn2-core 12 k 2023-01-11T21:08:36.9407157Z Installing for dependencies: 2023-01-11T21:08:36.9407564Z apr x86_64 1.7.0-9.amzn2 amzn2-core 122 k 2023-01-11T21:08:36.9408104Z apr-util x86_64 1.6.1-5.amzn2.0.2 amzn2-core 99 k 2023-01-11T21:08:36.9408528Z apr-util-bdb x86_64 1.6.1-5.amzn2.0.2 amzn2-core 19 k 2023-01-11T21:08:36.9408970Z avahi-libs x86_64 0.6.31-20.amzn2 amzn2-core 61 k 2023-01-11T21:08:36.9409439Z cpp x86_64 7.3.1-15.amzn2 amzn2-core 9.2 M 2023-01-11T21:08:36.9409858Z dwz x86_64 0.11-3.amzn2.0.3 amzn2-core 98 k 2023-01-11T21:08:36.9410637Z efivar-libs x86_64 31-4.amzn2.0.4 amzn2-core 68 k 2023-01-11T21:08:36.9411113Z elfutils-libelf-devel x86_64 0.176-2.amzn2 amzn2-core 40 k 2023-01-11T21:08:36.9411581Z emacs-filesystem noarch 1:27.2-4.amzn2.0.1 amzn2-core 67 k 2023-01-11T21:08:36.9412023Z gdb x86_64 8.0.1-36.amzn2.0.1 amzn2-core 3.1 M 2023-01-11T21:08:36.9412542Z gettext-common-devel noarch 0.19.8.1-3.amzn2 amzn2-core 410 k 2023-01-11T21:08:36.9413031Z gettext-devel x86_64 0.19.8.1-3.amzn2 amzn2-core 320 k 2023-01-11T21:08:36.9413477Z glibc-devel x86_64 2.26-62.amzn2 amzn2-core 995 k 2023-01-11T21:08:36.9413904Z glibc-headers x86_64 2.26-62.amzn2 amzn2-core 516 k 2023-01-11T21:08:36.9414340Z gnutls x86_64 3.3.29-9.amzn2.0.1 amzn2-core 661 k 2023-01-11T21:08:36.9414775Z go-srpm-macros noarch 3.0.15-23.amzn2.0.2 amzn2-core 23 k 2023-01-11T21:08:36.9415224Z kernel-devel x86_64 4.14.301-224.520.amzn2 amzn2-core 13 M 2023-01-11T21:08:36.9415649Z kernel-headers x86_64 4.14.301-224.520.amzn2 amzn2-core 1.2 M 2023-01-11T21:08:36.9416081Z libatomic x86_64 7.3.1-15.amzn2 amzn2-core 46 k 2023-01-11T21:08:36.9416512Z libcilkrts x86_64 7.3.1-15.amzn2 amzn2-core 85 k 2023-01-11T21:08:36.9416925Z libgfortran x86_64 7.3.1-15.amzn2 amzn2-core 536 k 2023-01-11T21:08:36.9417349Z libitm x86_64 7.3.1-15.amzn2 amzn2-core 85 k 2023-01-11T21:08:36.9417773Z libmodman x86_64 2.0.1-8.amzn2.0.2 amzn2-core 29 k 2023-01-11T21:08:36.9418200Z libmpc x86_64 1.0.1-3.amzn2.0.2 amzn2-core 52 k 2023-01-11T21:08:36.9418606Z libmpx x86_64 7.3.1-15.amzn2 amzn2-core 51 k 2023-01-11T21:08:36.9419026Z libproxy x86_64 0.4.11-10.amzn2.0.3 amzn2-core 61 k 2023-01-11T21:08:36.9419455Z libquadmath x86_64 7.3.1-15.amzn2 amzn2-core 189 k 2023-01-11T21:08:36.9419864Z libsanitizer x86_64 7.3.1-15.amzn2 amzn2-core 642 k 2023-01-11T21:08:36.9420283Z m4 x86_64 1.4.16-10.amzn2.0.2 amzn2-core 256 k 2023-01-11T21:08:36.9420703Z mokutil x86_64 1:0.3.0-10.amzn2.0.1 amzn2-core 39 k 2023-01-11T21:08:36.9421118Z mpfr x86_64 3.1.1-4.amzn2.0.2 amzn2-core 208 k 2023-01-11T21:08:36.9421513Z neon x86_64 0.30.0-3.amzn2.0.2 amzn2-core 166 k 2023-01-11T21:08:36.9421933Z pakchois x86_64 0.4-10.amzn2.0.2 amzn2-core 14 k 2023-01-11T21:08:36.9422379Z perl-Data-Dumper x86_64 2.145-3.amzn2.0.2 amzn2-core 48 k 2023-01-11T21:08:36.9422822Z perl-Test-Harness noarch 3.28-3.amzn2 amzn2-core 302 k 2023-01-11T21:08:36.9423287Z perl-Thread-Queue noarch 3.02-2.amzn2 amzn2-core 17 k 2023-01-11T21:08:36.9423752Z perl-XML-Parser x86_64 2.41-10.amzn2.0.2 amzn2-core 223 k 2023-01-11T21:08:36.9424218Z perl-srpm-macros noarch 1-8.amzn2.0.1 amzn2-core 4.7 k 2023-01-11T21:08:36.9424672Z subversion-libs x86_64 1.7.14-16.amzn2.0.1 amzn2-core 912 k 2023-01-11T21:08:36.9425210Z systemtap-client x86_64 4.5-1.amzn2.0.1 amzn2-core 3.7 M 2023-01-11T21:08:36.9425667Z systemtap-devel x86_64 4.5-1.amzn2.0.1 amzn2-core 2.3 M 2023-01-11T21:08:36.9426090Z trousers x86_64 0.3.14-2.amzn2.0.2 amzn2-core 294 k 2023-01-11T21:08:36.9426519Z zlib-devel x86_64 1.2.7-19.amzn2.0.2 amzn2-core 50 k 2023-01-11T21:08:36.9426722Z 2023-01-11T21:08:36.9426836Z Transaction Summary 2023-01-11T21:08:36.9427116Z ================================================================================ 2023-01-11T21:08:36.9427408Z Install 25 Packages (+43 Dependent packages) 2023-01-11T21:08:36.9427604Z 2023-01-11T21:08:36.9427736Z Total download size: 96 M 2023-01-11T21:08:36.9428000Z Installed size: 303 M 2023-01-11T21:08:36.9428245Z Downloading packages: 2023-01-11T21:08:36.9446078Z Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2023-01-11T21:08:38.3837403Z -------------------------------------------------------------------------------- 2023-01-11T21:08:38.3837885Z Total 67 MB/s | 96 MB 00:01 2023-01-11T21:08:38.4892283Z Running transaction check 2023-01-11T21:08:38.5655177Z Running transaction test 2023-01-11T21:08:40.7552318Z Transaction test succeeded 2023-01-11T21:08:40.7555293Z Running transaction 2023-01-11T21:08:45.9741398Z Installing : mpfr-3.1.1-4.amzn2.0.2.x86_64 1/68 2023-01-11T21:08:48.4360203Z Installing : libmpc-1.0.1-3.amzn2.0.2.x86_64 2/68 2023-01-11T21:08:51.0886083Z Installing : m4-1.4.16-10.amzn2.0.2.x86_64 3/68 2023-01-11T21:08:53.5522518Z Installing : apr-1.7.0-9.amzn2.x86_64 4/68 2023-01-11T21:08:56.0040192Z Installing : apr-util-bdb-1.6.1-5.amzn2.0.2.x86_64 5/68 2023-01-11T21:08:58.4750615Z Installing : apr-util-1.6.1-5.amzn2.0.2.x86_64 6/68 2023-01-11T21:09:00.5254963Z Installing : avahi-libs-0.6.31-20.amzn2.x86_64 7/68 2023-01-11T21:09:00.5721282Z Installing : libquadmath-7.3.1-15.amzn2.x86_64 8/68 2023-01-11T21:09:00.5946937Z Installing : patch-2.7.1-12.amzn2.0.2.x86_64 9/68 2023-01-11T21:09:00.6804404Z Installing : perl-Thread-Queue-3.02-2.amzn2.noarch 10/68 2023-01-11T21:09:01.7406068Z Installing : libgfortran-7.3.1-15.amzn2.x86_64 11/68 2023-01-11T21:09:01.7793604Z Installing : cpp-7.3.1-15.amzn2.x86_64 12/68 2023-01-11T21:09:01.8125894Z Installing : libmodman-2.0.1-8.amzn2.0.2.x86_64 13/68 2023-01-11T21:09:01.8720933Z Installing : libproxy-0.4.11-10.amzn2.0.3.x86_64 14/68 2023-01-11T21:09:01.9293667Z Installing : perl-XML-Parser-2.41-10.amzn2.0.2.x86_64 15/68 2023-01-11T21:09:02.0411465Z Installing : elfutils-0.176-2.amzn2.x86_64 16/68 2023-01-11T21:09:02.0724478Z Installing : libsanitizer-7.3.1-15.amzn2.x86_64 17/68 2023-01-11T21:09:02.0960865Z Installing : 1:emacs-filesystem-27.2-4.amzn2.0.1.noarch 18/68 2023-01-11T21:09:02.1281693Z Installing : efivar-libs-31-4.amzn2.0.4.x86_64 19/68 2023-01-11T21:09:02.1586657Z Installing : 1:mokutil-0.3.0-10.amzn2.0.1.x86_64 20/68 2023-01-11T21:09:02.2395393Z Installing : gettext-common-devel-0.19.8.1-3.amzn2.noarch 21/68 2023-01-11T21:09:02.2922527Z Installing : gettext-devel-0.19.8.1-3.amzn2.x86_64 22/68 2023-01-11T21:09:02.3798957Z Installing : dwz-0.11-3.amzn2.0.3.x86_64 23/68 2023-01-11T21:09:02.5486866Z Installing : trousers-0.3.14-2.amzn2.0.2.x86_64 24/68 2023-01-11T21:09:02.5847732Z Installing : gnutls-3.3.29-9.amzn2.0.1.x86_64 25/68 2023-01-11T21:09:02.9821793Z Installing : libitm-7.3.1-15.amzn2.x86_64 26/68 2023-01-11T21:09:03.0111409Z Installing : gdb-8.0.1-36.amzn2.0.1.x86_64 27/68 2023-01-11T21:09:03.0402878Z Installing : libmpx-7.3.1-15.amzn2.x86_64 28/68 2023-01-11T21:09:03.0599861Z Installing : perl-srpm-macros-1-8.amzn2.0.1.noarch 29/68 2023-01-11T21:09:03.0907305Z Installing : go-srpm-macros-3.0.15-23.amzn2.0.2.noarch 30/68 2023-01-11T21:09:03.1183762Z Installing : system-rpm-config-9.1.0-76.amzn2.0.14.noarch 31/68 2023-01-11T21:09:03.2084703Z Installing : perl-Data-Dumper-2.145-3.amzn2.0.2.x86_64 32/68 2023-01-11T21:09:03.2933319Z Installing : autoconf-2.69-11.amzn2.noarch 33/68 2023-01-11T21:09:03.3956975Z Installing : perl-Test-Harness-3.28-3.amzn2.noarch 34/68 2023-01-11T21:09:03.4566137Z Installing : automake-1.13.4-3.1.amzn2.noarch 35/68 2023-01-11T21:09:03.4774058Z Installing : zlib-devel-1.2.7-19.amzn2.0.2.x86_64 36/68 2023-01-11T21:09:03.4975498Z Installing : elfutils-libelf-devel-0.176-2.amzn2.x86_64 37/68 2023-01-11T21:09:03.7892404Z Installing : libatomic-7.3.1-15.amzn2.x86_64 38/68 2023-01-11T21:09:03.9571538Z Installing : kernel-headers-4.14.301-224.520.amzn2.x86_64 39/68 2023-01-11T21:09:04.0896756Z Installing : glibc-headers-2.26-62.amzn2.x86_64 40/68 2023-01-11T21:09:04.1263259Z Installing : glibc-devel-2.26-62.amzn2.x86_64 41/68 2023-01-11T21:09:06.2691318Z Installing : libcilkrts-7.3.1-15.amzn2.x86_64 42/68 2023-01-11T21:09:10.0792127Z Installing : gcc-7.3.1-15.amzn2.x86_64 43/68 2023-01-11T21:09:21.2651608Z Installing : kernel-devel-4.14.301-224.520.amzn2.x86_64 44/68 2023-01-11T21:09:21.8814498Z Installing : systemtap-devel-4.5-1.amzn2.0.1.x86_64 45/68 2023-01-11T21:09:21.9415968Z Installing : systemtap-client-4.5-1.amzn2.0.1.x86_64 46/68 2023-01-11T21:09:21.9990203Z Installing : pakchois-0.4-10.amzn2.0.2.x86_64 47/68 2023-01-11T21:09:22.1334746Z Installing : neon-0.30.0-3.amzn2.0.2.x86_64 48/68 2023-01-11T21:09:22.3137534Z Installing : subversion-libs-1.7.14-16.amzn2.0.1.x86_64 49/68 2023-01-11T21:09:22.4161072Z Installing : subversion-1.7.14-16.amzn2.0.1.x86_64 50/68 2023-01-11T21:09:23.6311264Z Installing : systemtap-4.5-1.amzn2.0.1.x86_64 51/68 2023-01-11T21:09:25.2518724Z Installing : gcc-gfortran-7.3.1-15.amzn2.x86_64 52/68 2023-01-11T21:09:25.3651927Z Installing : gcc-c++-7.3.1-15.amzn2.x86_64 53/68 2023-01-11T21:09:25.4109341Z Installing : libtool-2.4.2-22.2.amzn2.0.2.x86_64 54/68 2023-01-11T21:09:25.4527265Z Installing : intltool-0.50.2-7.amzn2.noarch 55/68 2023-01-11T21:09:25.5074573Z Installing : rpm-build-4.11.3-48.amzn2.0.2.x86_64 56/68 2023-01-11T21:09:25.5652698Z Installing : cscope-15.8-10.amzn2.0.2.x86_64 57/68 2023-01-11T21:09:25.6672595Z Installing : flex-2.5.37-3.amzn2.0.3.x86_64 58/68 2023-01-11T21:09:25.7300982Z Installing : bison-3.0.4-6.amzn2.0.2.x86_64 59/68 2023-01-11T21:09:25.7771819Z Installing : rcs-5.9.0-5.amzn2.0.2.x86_64 60/68 2023-01-11T21:09:25.8146280Z Installing : ctags-5.8-13.amzn2.0.2.x86_64 61/68 2023-01-11T21:09:25.8594701Z Installing : indent-2.2.11-13.amzn2.0.2.x86_64 62/68 2023-01-11T21:09:26.5478305Z Installing : patchutils-0.3.3-4.amzn2.0.1.x86_64 63/68 2023-01-11T21:09:26.5897032Z Installing : 1:doxygen-1.8.5-4.amzn2.x86_64 64/68 2023-01-11T21:09:26.6165506Z Installing : diffstat-1.57-4.amzn2.0.2.x86_64 65/68 2023-01-11T21:09:26.9262482Z Installing : byacc-1.9.20130304-3.amzn2.0.2.x86_64 66/68 2023-01-11T21:09:26.9673755Z Installing : swig-3.0.12-11.amzn2.0.3.x86_64 67/68 2023-01-11T21:09:27.0301626Z Installing : rpm-sign-4.11.3-48.amzn2.0.2.x86_64 68/68 2023-01-11T21:09:27.0438768Z Verifying : elfutils-libelf-devel-0.176-2.amzn2.x86_64 1/68 2023-01-11T21:09:27.0542000Z Verifying : perl-Thread-Queue-3.02-2.amzn2.noarch 2/68 2023-01-11T21:09:27.0639168Z Verifying : gettext-devel-0.19.8.1-3.amzn2.x86_64 3/68 2023-01-11T21:09:27.0736185Z Verifying : patch-2.7.1-12.amzn2.0.2.x86_64 4/68 2023-01-11T21:09:27.0837736Z Verifying : flex-2.5.37-3.amzn2.0.3.x86_64 5/68 2023-01-11T21:09:27.0925845Z Verifying : pakchois-0.4-10.amzn2.0.2.x86_64 6/68 2023-01-11T21:09:27.1031209Z Verifying : rpm-sign-4.11.3-48.amzn2.0.2.x86_64 7/68 2023-01-11T21:09:27.1126015Z Verifying : glibc-devel-2.26-62.amzn2.x86_64 8/68 2023-01-11T21:09:27.1217784Z Verifying : gcc-gfortran-7.3.1-15.amzn2.x86_64 9/68 2023-01-11T21:09:27.1315290Z Verifying : swig-3.0.12-11.amzn2.0.3.x86_64 10/68 2023-01-11T21:09:27.1415025Z Verifying : byacc-1.9.20130304-3.amzn2.0.2.x86_64 11/68 2023-01-11T21:09:27.1498043Z Verifying : libmpc-1.0.1-3.amzn2.0.2.x86_64 12/68 2023-01-11T21:09:27.1602573Z Verifying : libcilkrts-7.3.1-15.amzn2.x86_64 13/68 2023-01-11T21:09:27.1715582Z Verifying : kernel-headers-4.14.301-224.520.amzn2.x86_64 14/68 2023-01-11T21:09:27.1812422Z Verifying : libproxy-0.4.11-10.amzn2.0.3.x86_64 15/68 2023-01-11T21:09:27.1915463Z Verifying : cscope-15.8-10.amzn2.0.2.x86_64 16/68 2023-01-11T21:09:27.2017472Z Verifying : diffstat-1.57-4.amzn2.0.2.x86_64 17/68 2023-01-11T21:09:27.2125220Z Verifying : 1:doxygen-1.8.5-4.amzn2.x86_64 18/68 2023-01-11T21:09:27.2233472Z Verifying : gcc-c++-7.3.1-15.amzn2.x86_64 19/68 2023-01-11T21:09:27.2330093Z Verifying : libatomic-7.3.1-15.amzn2.x86_64 20/68 2023-01-11T21:09:27.2421329Z Verifying : system-rpm-config-9.1.0-76.amzn2.0.14.noarch 21/68 2023-01-11T21:09:27.2516593Z Verifying : systemtap-devel-4.5-1.amzn2.0.1.x86_64 22/68 2023-01-11T21:09:27.2607801Z Verifying : zlib-devel-1.2.7-19.amzn2.0.2.x86_64 23/68 2023-01-11T21:09:27.2702332Z Verifying : glibc-headers-2.26-62.amzn2.x86_64 24/68 2023-01-11T21:09:27.2814004Z Verifying : perl-Test-Harness-3.28-3.amzn2.noarch 25/68 2023-01-11T21:09:27.2911419Z Verifying : autoconf-2.69-11.amzn2.noarch 26/68 2023-01-11T21:09:27.3004993Z Verifying : libquadmath-7.3.1-15.amzn2.x86_64 27/68 2023-01-11T21:09:27.3109825Z Verifying : intltool-0.50.2-7.amzn2.noarch 28/68 2023-01-11T21:09:27.3231847Z Verifying : apr-util-1.6.1-5.amzn2.0.2.x86_64 29/68 2023-01-11T21:09:27.3336350Z Verifying : cpp-7.3.1-15.amzn2.x86_64 30/68 2023-01-11T21:09:27.3451643Z Verifying : rpm-build-4.11.3-48.amzn2.0.2.x86_64 31/68 2023-01-11T21:09:27.3555020Z Verifying : go-srpm-macros-3.0.15-23.amzn2.0.2.noarch 32/68 2023-01-11T21:09:27.3646349Z Verifying : perl-Data-Dumper-2.145-3.amzn2.0.2.x86_64 33/68 2023-01-11T21:09:27.3741620Z Verifying : perl-srpm-macros-1-8.amzn2.0.1.noarch 34/68 2023-01-11T21:09:27.3837235Z Verifying : gnutls-3.3.29-9.amzn2.0.1.x86_64 35/68 2023-01-11T21:09:27.3933575Z Verifying : subversion-libs-1.7.14-16.amzn2.0.1.x86_64 36/68 2023-01-11T21:09:27.4016975Z Verifying : automake-1.13.4-3.1.amzn2.noarch 37/68 2023-01-11T21:09:27.4113715Z Verifying : apr-util-bdb-1.6.1-5.amzn2.0.2.x86_64 38/68 2023-01-11T21:09:27.4197055Z Verifying : libmpx-7.3.1-15.amzn2.x86_64 39/68 2023-01-11T21:09:27.4286100Z Verifying : avahi-libs-0.6.31-20.amzn2.x86_64 40/68 2023-01-11T21:09:27.4387586Z Verifying : bison-3.0.4-6.amzn2.0.2.x86_64 41/68 2023-01-11T21:09:27.4471609Z Verifying : libgfortran-7.3.1-15.amzn2.x86_64 42/68 2023-01-11T21:09:27.4601341Z Verifying : gdb-8.0.1-36.amzn2.0.1.x86_64 43/68 2023-01-11T21:09:27.4693685Z Verifying : patchutils-0.3.3-4.amzn2.0.1.x86_64 44/68 2023-01-11T21:09:27.4786753Z Verifying : libitm-7.3.1-15.amzn2.x86_64 45/68 2023-01-11T21:09:27.4880147Z Verifying : libtool-2.4.2-22.2.amzn2.0.2.x86_64 46/68 2023-01-11T21:09:27.4973985Z Verifying : gcc-7.3.1-15.amzn2.x86_64 47/68 2023-01-11T21:09:27.5077710Z Verifying : indent-2.2.11-13.amzn2.0.2.x86_64 48/68 2023-01-11T21:09:27.5169637Z Verifying : subversion-1.7.14-16.amzn2.0.1.x86_64 49/68 2023-01-11T21:09:27.5276950Z Verifying : apr-1.7.0-9.amzn2.x86_64 50/68 2023-01-11T21:09:27.5381476Z Verifying : ctags-5.8-13.amzn2.0.2.x86_64 51/68 2023-01-11T21:09:27.5472403Z Verifying : 1:mokutil-0.3.0-10.amzn2.0.1.x86_64 52/68 2023-01-11T21:09:27.5563201Z Verifying : mpfr-3.1.1-4.amzn2.0.2.x86_64 53/68 2023-01-11T21:09:27.5679365Z Verifying : trousers-0.3.14-2.amzn2.0.2.x86_64 54/68 2023-01-11T21:09:27.5791583Z Verifying : neon-0.30.0-3.amzn2.0.2.x86_64 55/68 2023-01-11T21:09:27.5875862Z Verifying : systemtap-4.5-1.amzn2.0.1.x86_64 56/68 2023-01-11T21:09:27.5959914Z Verifying : dwz-0.11-3.amzn2.0.3.x86_64 57/68 2023-01-11T21:09:27.6052354Z Verifying : gettext-common-devel-0.19.8.1-3.amzn2.noarch 58/68 2023-01-11T21:09:27.6154693Z Verifying : systemtap-client-4.5-1.amzn2.0.1.x86_64 59/68 2023-01-11T21:09:27.6243268Z Verifying : efivar-libs-31-4.amzn2.0.4.x86_64 60/68 2023-01-11T21:09:27.6334310Z Verifying : rcs-5.9.0-5.amzn2.0.2.x86_64 61/68 2023-01-11T21:09:27.6483087Z Verifying : kernel-devel-4.14.301-224.520.amzn2.x86_64 62/68 2023-01-11T21:09:27.6596282Z Verifying : 1:emacs-filesystem-27.2-4.amzn2.0.1.noarch 63/68 2023-01-11T21:09:27.6689863Z Verifying : libsanitizer-7.3.1-15.amzn2.x86_64 64/68 2023-01-11T21:09:27.6791734Z Verifying : elfutils-0.176-2.amzn2.x86_64 65/68 2023-01-11T21:09:27.6881575Z Verifying : m4-1.4.16-10.amzn2.0.2.x86_64 66/68 2023-01-11T21:09:27.6959957Z Verifying : perl-XML-Parser-2.41-10.amzn2.0.2.x86_64 67/68 2023-01-11T21:09:27.7695072Z Verifying : libmodman-2.0.1-8.amzn2.0.2.x86_64 68/68 2023-01-11T21:09:27.7697003Z 2023-01-11T21:09:27.7697544Z Installed: 2023-01-11T21:09:27.7697991Z autoconf.noarch 0:2.69-11.amzn2 2023-01-11T21:09:27.7698431Z automake.noarch 0:1.13.4-3.1.amzn2 2023-01-11T21:09:27.7700737Z bison.x86_64 0:3.0.4-6.amzn2.0.2 2023-01-11T21:09:27.7704049Z byacc.x86_64 0:1.9.20130304-3.amzn2.0.2 2023-01-11T21:09:27.7704499Z cscope.x86_64 0:15.8-10.amzn2.0.2 2023-01-11T21:09:27.7704938Z ctags.x86_64 0:5.8-13.amzn2.0.2 2023-01-11T21:09:27.7705365Z diffstat.x86_64 0:1.57-4.amzn2.0.2 2023-01-11T21:09:27.7705762Z doxygen.x86_64 1:1.8.5-4.amzn2 2023-01-11T21:09:27.7706388Z elfutils.x86_64 0:0.176-2.amzn2 2023-01-11T21:09:27.7706830Z flex.x86_64 0:2.5.37-3.amzn2.0.3 2023-01-11T21:09:27.7707235Z gcc.x86_64 0:7.3.1-15.amzn2 2023-01-11T21:09:27.7707631Z gcc-c++.x86_64 0:7.3.1-15.amzn2 2023-01-11T21:09:27.7708050Z gcc-gfortran.x86_64 0:7.3.1-15.amzn2 2023-01-11T21:09:27.7708470Z indent.x86_64 0:2.2.11-13.amzn2.0.2 2023-01-11T21:09:27.7708868Z intltool.noarch 0:0.50.2-7.amzn2 2023-01-11T21:09:27.7709305Z libtool.x86_64 0:2.4.2-22.2.amzn2.0.2 2023-01-11T21:09:27.7709728Z patch.x86_64 0:2.7.1-12.amzn2.0.2 2023-01-11T21:09:27.7710159Z patchutils.x86_64 0:0.3.3-4.amzn2.0.1 2023-01-11T21:09:27.7710555Z rcs.x86_64 0:5.9.0-5.amzn2.0.2 2023-01-11T21:09:27.7710980Z rpm-build.x86_64 0:4.11.3-48.amzn2.0.2 2023-01-11T21:09:27.7711406Z rpm-sign.x86_64 0:4.11.3-48.amzn2.0.2 2023-01-11T21:09:27.7711821Z subversion.x86_64 0:1.7.14-16.amzn2.0.1 2023-01-11T21:09:27.7712213Z swig.x86_64 0:3.0.12-11.amzn2.0.3 2023-01-11T21:09:27.7712660Z system-rpm-config.noarch 0:9.1.0-76.amzn2.0.14 2023-01-11T21:09:27.7713108Z systemtap.x86_64 0:4.5-1.amzn2.0.1 2023-01-11T21:09:27.7713306Z 2023-01-11T21:09:27.7713414Z Dependency Installed: 2023-01-11T21:09:27.7713804Z apr.x86_64 0:1.7.0-9.amzn2 2023-01-11T21:09:27.7714230Z apr-util.x86_64 0:1.6.1-5.amzn2.0.2 2023-01-11T21:09:27.7714659Z apr-util-bdb.x86_64 0:1.6.1-5.amzn2.0.2 2023-01-11T21:09:27.7715073Z avahi-libs.x86_64 0:0.6.31-20.amzn2 2023-01-11T21:09:27.7715504Z cpp.x86_64 0:7.3.1-15.amzn2 2023-01-11T21:09:27.7715910Z dwz.x86_64 0:0.11-3.amzn2.0.3 2023-01-11T21:09:27.7716310Z efivar-libs.x86_64 0:31-4.amzn2.0.4 2023-01-11T21:09:27.7716758Z elfutils-libelf-devel.x86_64 0:0.176-2.amzn2 2023-01-11T21:09:27.7717211Z emacs-filesystem.noarch 1:27.2-4.amzn2.0.1 2023-01-11T21:09:27.7717640Z gdb.x86_64 0:8.0.1-36.amzn2.0.1 2023-01-11T21:09:27.7718066Z gettext-common-devel.noarch 0:0.19.8.1-3.amzn2 2023-01-11T21:09:27.7718607Z gettext-devel.x86_64 0:0.19.8.1-3.amzn2 2023-01-11T21:09:27.7719035Z glibc-devel.x86_64 0:2.26-62.amzn2 2023-01-11T21:09:27.7719443Z glibc-headers.x86_64 0:2.26-62.amzn2 2023-01-11T21:09:27.7719872Z gnutls.x86_64 0:3.3.29-9.amzn2.0.1 2023-01-11T21:09:27.7720298Z go-srpm-macros.noarch 0:3.0.15-23.amzn2.0.2 2023-01-11T21:09:27.7720756Z kernel-devel.x86_64 0:4.14.301-224.520.amzn2 2023-01-11T21:09:27.7721173Z kernel-headers.x86_64 0:4.14.301-224.520.amzn2 2023-01-11T21:09:27.7721605Z libatomic.x86_64 0:7.3.1-15.amzn2 2023-01-11T21:09:27.7722076Z libcilkrts.x86_64 0:7.3.1-15.amzn2 2023-01-11T21:09:27.7722512Z libgfortran.x86_64 0:7.3.1-15.amzn2 2023-01-11T21:09:27.7722905Z libitm.x86_64 0:7.3.1-15.amzn2 2023-01-11T21:09:27.7723312Z libmodman.x86_64 0:2.0.1-8.amzn2.0.2 2023-01-11T21:09:27.7723725Z libmpc.x86_64 0:1.0.1-3.amzn2.0.2 2023-01-11T21:09:27.7724111Z libmpx.x86_64 0:7.3.1-15.amzn2 2023-01-11T21:09:27.7724517Z libproxy.x86_64 0:0.4.11-10.amzn2.0.3 2023-01-11T21:09:27.7724943Z libquadmath.x86_64 0:7.3.1-15.amzn2 2023-01-11T21:09:27.7725366Z libsanitizer.x86_64 0:7.3.1-15.amzn2 2023-01-11T21:09:27.7725745Z m4.x86_64 0:1.4.16-10.amzn2.0.2 2023-01-11T21:09:27.7726149Z mokutil.x86_64 1:0.3.0-10.amzn2.0.1 2023-01-11T21:09:27.7726559Z mpfr.x86_64 0:3.1.1-4.amzn2.0.2 2023-01-11T21:09:27.7726945Z neon.x86_64 0:0.30.0-3.amzn2.0.2 2023-01-11T21:09:27.7727351Z pakchois.x86_64 0:0.4-10.amzn2.0.2 2023-01-11T21:09:27.7727782Z perl-Data-Dumper.x86_64 0:2.145-3.amzn2.0.2 2023-01-11T21:09:27.7728235Z perl-Test-Harness.noarch 0:3.28-3.amzn2 2023-01-11T21:09:27.7728681Z perl-Thread-Queue.noarch 0:3.02-2.amzn2 2023-01-11T21:09:27.7729135Z perl-XML-Parser.x86_64 0:2.41-10.amzn2.0.2 2023-01-11T21:09:27.7729592Z perl-srpm-macros.noarch 0:1-8.amzn2.0.1 2023-01-11T21:09:27.7730044Z subversion-libs.x86_64 0:1.7.14-16.amzn2.0.1 2023-01-11T21:09:27.7731121Z systemtap-client.x86_64 0:4.5-1.amzn2.0.1 2023-01-11T21:09:27.7731566Z systemtap-devel.x86_64 0:4.5-1.amzn2.0.1 2023-01-11T21:09:27.7731988Z trousers.x86_64 0:0.3.14-2.amzn2.0.2 2023-01-11T21:09:27.7732385Z zlib-devel.x86_64 0:1.2.7-19.amzn2.0.2 2023-01-11T21:09:27.7732582Z 2023-01-11T21:09:27.7732687Z Complete! 2023-01-11T21:09:27.8096389Z ++ uname -r 2023-01-11T21:09:27.8105540Z + sudo yum install -y 'kernel-devel-uname-r == 4.14.252-195.483.amzn2.x86_64' 2023-01-11T21:09:28.3168203Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2023-01-11T21:09:28.3279678Z Existing lock /var/run/yum.pid: another copy is running as pid 35522. 2023-01-11T21:09:28.3280104Z Another app is currently holding the yum lock; waiting for it to exit... 2023-01-11T21:09:28.3288478Z The other application is: yum 2023-01-11T21:09:28.3288777Z Memory : 92 M RSS (310 MB VSZ) 2023-01-11T21:09:28.3289846Z Started: Wed Jan 11 21:09:27 2023 - 00:01 ago 2023-01-11T21:09:28.3290432Z State : Running, pid: 35522 2023-01-11T21:09:30.3315049Z Another app is currently holding the yum lock; waiting for it to exit... 2023-01-11T21:09:30.3321508Z The other application is: yum 2023-01-11T21:09:30.3321789Z Memory : 169 M RSS (387 MB VSZ) 2023-01-11T21:09:30.3322761Z Started: Wed Jan 11 21:09:27 2023 - 00:03 ago 2023-01-11T21:09:30.3323258Z State : Running, pid: 35522 2023-01-11T21:09:32.6098690Z Resolving Dependencies 2023-01-11T21:09:32.6104754Z --> Running transaction check 2023-01-11T21:09:32.6105240Z ---> Package kernel-devel.x86_64 0:4.14.252-195.483.amzn2 will be installed 2023-01-11T21:09:32.8957560Z --> Finished Dependency Resolution 2023-01-11T21:09:32.9741570Z 2023-01-11T21:09:32.9741938Z Dependencies Resolved 2023-01-11T21:09:32.9747853Z 2023-01-11T21:09:32.9748501Z ================================================================================ 2023-01-11T21:09:32.9748913Z Package Arch Version Repository Size 2023-01-11T21:09:32.9749271Z ================================================================================ 2023-01-11T21:09:32.9749520Z Installing: 2023-01-11T21:09:32.9750029Z kernel-devel x86_64 4.14.252-195.483.amzn2 amzn2-core 13 M 2023-01-11T21:09:32.9750244Z 2023-01-11T21:09:32.9750361Z Transaction Summary 2023-01-11T21:09:32.9750638Z ================================================================================ 2023-01-11T21:09:32.9750889Z Install 1 Package 2023-01-11T21:09:32.9751046Z 2023-01-11T21:09:32.9751180Z Total download size: 13 M 2023-01-11T21:09:32.9751446Z Installed size: 60 M 2023-01-11T21:09:32.9751688Z Downloading packages: 2023-01-11T21:09:32.9760245Z Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2023-01-11T21:09:33.2784387Z Running transaction check 2023-01-11T21:09:33.2975230Z Running transaction test 2023-01-11T21:09:33.6988213Z Transaction test succeeded 2023-01-11T21:09:33.6991815Z Running transaction 2023-01-11T21:09:48.8381463Z Installing : kernel-devel-4.14.252-195.483.amzn2.x86_64 1/1 2023-01-11T21:09:48.9180078Z Verifying : kernel-devel-4.14.252-195.483.amzn2.x86_64 1/1 2023-01-11T21:09:48.9180381Z 2023-01-11T21:09:48.9180490Z Installed: 2023-01-11T21:09:48.9180886Z kernel-devel.x86_64 0:4.14.252-195.483.amzn2 2023-01-11T21:09:48.9181095Z 2023-01-11T21:09:48.9181207Z Complete! 2023-01-11T21:09:48.9506181Z + sudo modprobe backlight 2023-01-11T21:09:48.9699177Z + sudo curl -fsL -o /tmp/nvidia_driver https://s3.amazonaws.com/ossci-linux/nvidia_driver/NVIDIA-Linux-x86_64-515.76.run 2023-01-11T21:09:52.6513220Z + set +e 2023-01-11T21:09:52.6513741Z + sudo /bin/bash /tmp/nvidia_driver -s --no-drm 2023-01-11T21:09:54.0621191Z Verifying archive integrity... OK 2023-01-11T21:10:20.8357039Z Uncompressing NVIDIA Accelerated Graphics Driver for Linux-x86_64 515.76................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................ 2023-01-11T21:10:20.9773923Z 2023-01-11T21:10:20.9774631Z WARNING: The nvidia-drm module will not be installed. As a result, DRM-KMS will not function with this installation of the NVIDIA driver. 2023-01-11T21:10:20.9774969Z 2023-01-11T21:10:34.0575855Z 2023-01-11T21:10:34.0577088Z WARNING: nvidia-installer was forced to guess the X library path '/usr/lib64' and X module path '/usr/lib64/xorg/modules'; these paths were not queryable from the system. If X fails to find the NVIDIA X driver module, please install the `pkg-config` utility and the X.Org SDK/development package for your distribution and reinstall the driver. 2023-01-11T21:10:34.0578754Z 2023-01-11T21:10:41.6588842Z + NVIDIA_INSTALLATION_STATUS=0 2023-01-11T21:10:41.6589191Z + RESET_GPU=0 2023-01-11T21:10:41.6589638Z + '[' 0 -ne 0 ']' 2023-01-11T21:10:41.6591014Z ++ command -v nvidia-smi 2023-01-11T21:10:41.6593547Z + '[' -x /usr/bin/nvidia-smi ']' 2023-01-11T21:10:41.6597524Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2023-01-11T21:10:46.7180438Z + INSTALLED_DRIVER_VERSION=515.76 2023-01-11T21:10:46.7180774Z + NVIDIA_SMI_STATUS=0 2023-01-11T21:10:46.7181180Z + '[' 0 -ne 0 ']' 2023-01-11T21:10:46.7181420Z + '[' 0 -eq 1 ']' 2023-01-11T21:10:46.7181718Z + sudo rm -fv /tmp/nvidia_driver 2023-01-11T21:10:46.7824888Z removed ‘/tmp/nvidia_driver’ 2023-01-11T21:10:46.7842498Z + set -e 2023-01-11T21:10:46.7845108Z + post_install_nvidia_driver_common 2023-01-11T21:10:46.7848110Z + sudo modprobe nvidia 2023-01-11T21:10:46.7966372Z + echo 'After installing NVIDIA driver' 2023-01-11T21:10:46.7966615Z + lspci 2023-01-11T21:10:46.7966865Z After installing NVIDIA driver 2023-01-11T21:10:46.8203684Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 2023-01-11T21:10:46.8204114Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2023-01-11T21:10:46.8204603Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 2023-01-11T21:10:46.8205005Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01) 2023-01-11T21:10:46.8205366Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 2023-01-11T21:10:46.8205728Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2023-01-11T21:10:46.8206148Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2023-01-11T21:10:46.8208178Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2023-01-11T21:10:46.8208608Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01) 2023-01-11T21:10:46.8208940Z + lsmod 2023-01-11T21:10:46.8226556Z Module Size Used by 2023-01-11T21:10:46.8227083Z nvidia 40808448 0 2023-01-11T21:10:46.8227368Z drm 425984 1 nvidia 2023-01-11T21:10:46.8227639Z i2c_core 77824 2 nvidia,drm 2023-01-11T21:10:46.8227890Z backlight 16384 0 2023-01-11T21:10:46.8228553Z xt_conntrack 16384 1 2023-01-11T21:10:46.8230191Z ipt_MASQUERADE 16384 1 2023-01-11T21:10:46.8230511Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2023-01-11T21:10:46.8230818Z nf_conntrack_netlink 49152 0 2023-01-11T21:10:46.8231118Z nfnetlink 16384 2 nf_conntrack_netlink 2023-01-11T21:10:46.8231398Z xfrm_user 45056 1 2023-01-11T21:10:46.8231645Z xfrm_algo 16384 1 xfrm_user 2023-01-11T21:10:46.8231913Z xt_addrtype 16384 2 2023-01-11T21:10:46.8232181Z iptable_filter 16384 1 2023-01-11T21:10:46.8232422Z iptable_nat 16384 1 2023-01-11T21:10:46.8232684Z nf_conntrack_ipv4 16384 3 2023-01-11T21:10:46.8232972Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2023-01-11T21:10:46.8233259Z nf_nat_ipv4 16384 1 iptable_nat 2023-01-11T21:10:46.8233574Z nf_nat 36864 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2023-01-11T21:10:46.8234045Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2023-01-11T21:10:46.8234629Z br_netfilter 24576 0 2023-01-11T21:10:46.8234904Z bridge 172032 1 br_netfilter 2023-01-11T21:10:46.8235200Z stp 16384 1 bridge 2023-01-11T21:10:46.8235478Z llc 16384 2 bridge,stp 2023-01-11T21:10:46.8235718Z overlay 86016 0 2023-01-11T21:10:46.8235969Z sunrpc 393216 1 2023-01-11T21:10:46.8236221Z dm_mirror 28672 0 2023-01-11T21:10:46.8236472Z dm_region_hash 20480 1 dm_mirror 2023-01-11T21:10:46.8236780Z dm_log 20480 2 dm_region_hash,dm_mirror 2023-01-11T21:10:46.8237079Z dm_mod 143360 2 dm_log,dm_mirror 2023-01-11T21:10:46.8237330Z dax 69632 1 dm_mod 2023-01-11T21:10:46.8237583Z sb_edac 24576 0 2023-01-11T21:10:46.8237839Z crc32_pclmul 16384 0 2023-01-11T21:10:46.8238253Z ghash_clmulni_intel 16384 0 2023-01-11T21:10:46.8238870Z pcbc 16384 0 2023-01-11T21:10:46.8239193Z aesni_intel 188416 0 2023-01-11T21:10:46.8239642Z aes_x86_64 20480 1 aesni_intel 2023-01-11T21:10:46.8239931Z ata_piix 36864 0 2023-01-11T21:10:46.8240289Z crypto_simd 16384 1 aesni_intel 2023-01-11T21:10:46.8240647Z glue_helper 16384 1 aesni_intel 2023-01-11T21:10:46.8240945Z pcc_cpufreq 16384 0 2023-01-11T21:10:46.8241324Z libata 266240 1 ata_piix 2023-01-11T21:10:46.8241696Z mousedev 24576 0 2023-01-11T21:10:46.8242041Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2023-01-11T21:10:46.8242428Z evdev 20480 3 2023-01-11T21:10:46.8242798Z scsi_mod 245760 1 libata 2023-01-11T21:10:46.8243073Z psmouse 32768 0 2023-01-11T21:10:46.8243417Z button 16384 0 2023-01-11T21:10:46.8243741Z ena 114688 0 2023-01-11T21:10:46.8244012Z xen_blkfront 49152 2 2023-01-11T21:10:46.8244394Z crc32c_intel 24576 0 2023-01-11T21:10:46.8244729Z autofs4 49152 2 2023-01-11T21:10:46.8244992Z + modinfo nvidia 2023-01-11T21:10:46.8245591Z filename: /lib/modules/4.14.252-195.483.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2023-01-11T21:10:46.8246047Z firmware: nvidia/515.76/gsp.bin 2023-01-11T21:10:46.8246468Z alias: char-major-195-* 2023-01-11T21:10:46.8246756Z version: 515.76 2023-01-11T21:10:46.8247085Z supported: external 2023-01-11T21:10:46.8247414Z license: NVIDIA 2023-01-11T21:10:46.8247749Z srcversion: 51FD9DD90150B35351AFFBB 2023-01-11T21:10:46.8248128Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2023-01-11T21:10:46.8248509Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2023-01-11T21:10:46.8248832Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2023-01-11T21:10:46.8249222Z depends: i2c-core,drm 2023-01-11T21:10:46.8249571Z retpoline: Y 2023-01-11T21:10:46.8249859Z name: nvidia 2023-01-11T21:10:46.8250262Z vermagic: 4.14.252-195.483.amzn2.x86_64 SMP mod_unload modversions 2023-01-11T21:10:46.8251113Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2023-01-11T21:10:46.8251604Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2023-01-11T21:10:46.8251973Z parm: NVreg_ResmanDebugLevel:int 2023-01-11T21:10:46.8252317Z parm: NVreg_RmLogonRC:int 2023-01-11T21:10:46.8252694Z parm: NVreg_ModifyDeviceFiles:int 2023-01-11T21:10:46.8253004Z parm: NVreg_DeviceFileUID:int 2023-01-11T21:10:46.8253371Z parm: NVreg_DeviceFileGID:int 2023-01-11T21:10:46.8253715Z parm: NVreg_DeviceFileMode:int 2023-01-11T21:10:46.8254076Z parm: NVreg_InitializeSystemMemoryAllocations:int 2023-01-11T21:10:46.8254503Z parm: NVreg_UsePageAttributeTable:int 2023-01-11T21:10:46.8254875Z parm: NVreg_EnablePCIeGen3:int 2023-01-11T21:10:46.8255329Z parm: NVreg_EnableMSI:int 2023-01-11T21:10:46.8255666Z parm: NVreg_TCEBypassMode:int 2023-01-11T21:10:46.8256048Z parm: NVreg_EnableStreamMemOPs:int 2023-01-11T21:10:46.8256448Z parm: NVreg_RestrictProfilingToAdminUsers:int 2023-01-11T21:10:46.8256839Z parm: NVreg_PreserveVideoMemoryAllocations:int 2023-01-11T21:10:46.8257288Z parm: NVreg_EnableS0ixPowerManagement:int 2023-01-11T21:10:46.8257741Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2023-01-11T21:10:46.8258137Z parm: NVreg_DynamicPowerManagement:int 2023-01-11T21:10:46.8258586Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2023-01-11T21:10:46.8259054Z parm: NVreg_EnableGpuFirmware:int 2023-01-11T21:10:46.8259409Z parm: NVreg_EnableGpuFirmwareLogs:int 2023-01-11T21:10:46.8259859Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2023-01-11T21:10:46.8260357Z parm: NVreg_EnableUserNUMAManagement:int 2023-01-11T21:10:46.8260713Z parm: NVreg_MemoryPoolSize:int 2023-01-11T21:10:46.8261140Z parm: NVreg_KMallocHeapMaxSize:int 2023-01-11T21:10:46.8261533Z parm: NVreg_VMallocHeapMaxSize:int 2023-01-11T21:10:46.8261907Z parm: NVreg_IgnoreMMIOCheck:int 2023-01-11T21:10:46.8262219Z parm: NVreg_NvLinkDisable:int 2023-01-11T21:10:46.8262642Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2023-01-11T21:10:46.8263097Z parm: NVreg_RegisterPCIDriver:int 2023-01-11T21:10:46.8263475Z parm: NVreg_EnableDbgBreakpoint:int 2023-01-11T21:10:46.8263858Z parm: NVreg_RegistryDwords:charp 2023-01-11T21:10:46.8264269Z parm: NVreg_RegistryDwordsPerDevice:charp 2023-01-11T21:10:46.8296164Z parm: NVreg_RmMsg:charp 2023-01-11T21:10:46.8296690Z parm: NVreg_GpuBlacklist:charp 2023-01-11T21:10:46.8297114Z parm: NVreg_TemporaryFilePath:charp 2023-01-11T21:10:46.8297448Z parm: NVreg_ExcludedGpus:charp 2023-01-11T21:10:46.8297776Z parm: NVreg_DmaRemapPeerMmio:int 2023-01-11T21:10:46.8298073Z parm: rm_firmware_active:charp 2023-01-11T21:10:46.8298364Z + set +e 2023-01-11T21:10:46.8298670Z + nvidia-smi 2023-01-11T21:10:50.6178585Z Wed Jan 11 21:10:50 2023 2023-01-11T21:10:50.6179182Z +-----------------------------------------------------------------------------+ 2023-01-11T21:10:50.6179695Z | NVIDIA-SMI 515.76 Driver Version: 515.76 CUDA Version: 11.7 | 2023-01-11T21:10:50.6180170Z |-------------------------------+----------------------+----------------------+ 2023-01-11T21:10:50.6180684Z | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | 2023-01-11T21:10:50.6181182Z | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | 2023-01-11T21:10:50.6181519Z | | | MIG M. | 2023-01-11T21:10:50.6181852Z |===============================+======================+======================| 2023-01-11T21:10:50.6225730Z | 0 Tesla M60 Off | 00000000:00:1D.0 Off | 0 | 2023-01-11T21:10:50.6226093Z | N/A 25C P0 38W / 150W | 0MiB / 7680MiB | 0% Default | 2023-01-11T21:10:50.6226385Z | | | N/A | 2023-01-11T21:10:50.6226832Z +-------------------------------+----------------------+----------------------+ 2023-01-11T21:10:50.6273031Z | 1 Tesla M60 Off | 00000000:00:1E.0 Off | 0 | 2023-01-11T21:10:50.6273369Z | N/A 32C P0 37W / 150W | 0MiB / 7680MiB | 100% Default | 2023-01-11T21:10:50.6273681Z | | | N/A | 2023-01-11T21:10:50.6274116Z +-------------------------------+----------------------+----------------------+ 2023-01-11T21:10:50.6274504Z 2023-01-11T21:10:50.6275205Z +-----------------------------------------------------------------------------+ 2023-01-11T21:10:50.6275554Z | Processes: | 2023-01-11T21:10:50.6275886Z | GPU GI CI PID Type Process name GPU Memory | 2023-01-11T21:10:50.6276221Z | ID ID Usage | 2023-01-11T21:10:50.6276512Z |=============================================================================| 2023-01-11T21:10:50.6279638Z | No running processes found | 2023-01-11T21:10:50.6280102Z +-----------------------------------------------------------------------------+ 2023-01-11T21:10:51.1612347Z + NVIDIA_SMI_STATUS=0 2023-01-11T21:10:51.1612743Z + '[' 0 -eq 0 ']' 2023-01-11T21:10:51.1613060Z + echo 'INFO: Ignoring allowed status 0' 2023-01-11T21:10:51.1613343Z + set -e 2023-01-11T21:10:51.1613786Z INFO: Ignoring allowed status 0 2023-01-11T21:10:51.1620057Z == Installing nvidia container toolkit for amzn2 == 2023-01-11T21:10:51.1623821Z + sudo yum install -y yum-utils 2023-01-11T21:10:51.7010869Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2023-01-11T21:10:51.9771366Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version 2023-01-11T21:10:51.9771748Z Nothing to do 2023-01-11T21:10:51.9980304Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2023-01-11T21:10:52.5220954Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2023-01-11T21:10:52.5516240Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2023-01-11T21:10:52.5517036Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo 2023-01-11T21:10:52.5517564Z repo saved to /etc/yum.repos.d/nvidia-docker.repo 2023-01-11T21:10:52.5659244Z + sudo yum install -y nvidia-docker2 2023-01-11T21:10:53.0923880Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2023-01-11T21:10:53.1358513Z Retrieving key from https://nvidia.github.io/libnvidia-container/gpgkey 2023-01-11T21:10:53.1462390Z Importing GPG key 0xF796ECB0: 2023-01-11T21:10:53.1463079Z Userid : "NVIDIA CORPORATION (Open Source Projects) " 2023-01-11T21:10:53.1463922Z Fingerprint: c95b 321b 61e8 8c18 09c4 f759 ddca e044 f796 ecb0 2023-01-11T21:10:53.1464463Z From : https://nvidia.github.io/libnvidia-container/gpgkey 2023-01-11T21:10:53.5316156Z Retrieving key from https://nvidia.github.io/nvidia-container-runtime/gpgkey 2023-01-11T21:10:53.5406885Z Importing GPG key 0xF796ECB0: 2023-01-11T21:10:53.5407693Z Userid : "NVIDIA CORPORATION (Open Source Projects) " 2023-01-11T21:10:53.5408469Z Fingerprint: c95b 321b 61e8 8c18 09c4 f759 ddca e044 f796 ecb0 2023-01-11T21:10:53.5408962Z From : https://nvidia.github.io/nvidia-container-runtime/gpgkey 2023-01-11T21:10:53.7645437Z Retrieving key from https://nvidia.github.io/nvidia-docker/gpgkey 2023-01-11T21:10:53.7732598Z Importing GPG key 0xF796ECB0: 2023-01-11T21:10:53.7732989Z Userid : "NVIDIA CORPORATION (Open Source Projects) " 2023-01-11T21:10:53.7733395Z Fingerprint: c95b 321b 61e8 8c18 09c4 f759 ddca e044 f796 ecb0 2023-01-11T21:10:53.7733842Z From : https://nvidia.github.io/nvidia-docker/gpgkey 2023-01-11T21:10:55.4849394Z Resolving Dependencies 2023-01-11T21:10:55.4854883Z --> Running transaction check 2023-01-11T21:10:55.4855793Z ---> Package nvidia-docker2.noarch 0:2.11.0-1 will be installed 2023-01-11T21:10:55.4880564Z --> Processing Dependency: nvidia-container-toolkit >= 1.10.0-1 for package: nvidia-docker2-2.11.0-1.noarch 2023-01-11T21:10:55.5781473Z --> Running transaction check 2023-01-11T21:10:55.5781985Z ---> Package nvidia-container-toolkit.x86_64 0:1.11.0-1 will be installed 2023-01-11T21:10:55.5938762Z --> Processing Dependency: nvidia-container-toolkit-base = 1.11.0-1 for package: nvidia-container-toolkit-1.11.0-1.x86_64 2023-01-11T21:10:55.5949209Z --> Processing Dependency: libnvidia-container-tools < 2.0.0 for package: nvidia-container-toolkit-1.11.0-1.x86_64 2023-01-11T21:10:55.6075414Z --> Processing Dependency: libnvidia-container-tools >= 1.11.0-1 for package: nvidia-container-toolkit-1.11.0-1.x86_64 2023-01-11T21:10:55.6075906Z --> Running transaction check 2023-01-11T21:10:55.6076948Z ---> Package libnvidia-container-tools.x86_64 0:1.11.0-1 will be installed 2023-01-11T21:10:55.6087171Z --> Processing Dependency: libnvidia-container1(x86-64) >= 1.11.0-1 for package: libnvidia-container-tools-1.11.0-1.x86_64 2023-01-11T21:10:55.6114687Z --> Processing Dependency: libnvidia-container.so.1(NVC_1.0)(64bit) for package: libnvidia-container-tools-1.11.0-1.x86_64 2023-01-11T21:10:55.6115410Z --> Processing Dependency: libnvidia-container.so.1()(64bit) for package: libnvidia-container-tools-1.11.0-1.x86_64 2023-01-11T21:10:55.6116359Z ---> Package nvidia-container-toolkit-base.x86_64 0:1.11.0-1 will be installed 2023-01-11T21:10:55.6118842Z --> Running transaction check 2023-01-11T21:10:55.6119287Z ---> Package libnvidia-container1.x86_64 0:1.11.0-1 will be installed 2023-01-11T21:10:55.8504600Z --> Finished Dependency Resolution 2023-01-11T21:10:55.9248931Z 2023-01-11T21:10:55.9249246Z Dependencies Resolved 2023-01-11T21:10:55.9262622Z 2023-01-11T21:10:55.9263048Z ================================================================================ 2023-01-11T21:10:55.9263464Z Package Arch Version Repository Size 2023-01-11T21:10:55.9263819Z ================================================================================ 2023-01-11T21:10:55.9264092Z Installing: 2023-01-11T21:10:55.9264563Z nvidia-docker2 noarch 2.11.0-1 libnvidia-container 8.7 k 2023-01-11T21:10:55.9264929Z Installing for dependencies: 2023-01-11T21:10:55.9265392Z libnvidia-container-tools x86_64 1.11.0-1 libnvidia-container 49 k 2023-01-11T21:10:55.9265893Z libnvidia-container1 x86_64 1.11.0-1 libnvidia-container 1.0 M 2023-01-11T21:10:55.9266413Z nvidia-container-toolkit x86_64 1.11.0-1 libnvidia-container 780 k 2023-01-11T21:10:55.9266924Z nvidia-container-toolkit-base x86_64 1.11.0-1 libnvidia-container 2.5 M 2023-01-11T21:10:55.9267182Z 2023-01-11T21:10:55.9267291Z Transaction Summary 2023-01-11T21:10:55.9267564Z ================================================================================ 2023-01-11T21:10:55.9267881Z Install 1 Package (+4 Dependent packages) 2023-01-11T21:10:55.9268082Z 2023-01-11T21:10:55.9268192Z Total download size: 4.3 M 2023-01-11T21:10:55.9268437Z Installed size: 12 M 2023-01-11T21:10:55.9268700Z Downloading packages: 2023-01-11T21:10:56.0402283Z -------------------------------------------------------------------------------- 2023-01-11T21:10:56.0402731Z Total 38 MB/s | 4.3 MB 00:00 2023-01-11T21:10:56.0448302Z Running transaction check 2023-01-11T21:10:56.0621223Z Running transaction test 2023-01-11T21:10:56.0783847Z Transaction test succeeded 2023-01-11T21:10:56.0787184Z Running transaction 2023-01-11T21:10:56.5670199Z Installing : nvidia-container-toolkit-base-1.11.0-1.x86_64 1/5 2023-01-11T21:10:56.6066710Z Installing : libnvidia-container1-1.11.0-1.x86_64 2/5 2023-01-11T21:10:56.7139762Z Installing : libnvidia-container-tools-1.11.0-1.x86_64 3/5 2023-01-11T21:10:56.7393467Z Installing : nvidia-container-toolkit-1.11.0-1.x86_64 4/5 2023-01-11T21:10:56.7789993Z Installing : nvidia-docker2-2.11.0-1.noarch 5/5 2023-01-11T21:10:56.7926865Z Verifying : libnvidia-container1-1.11.0-1.x86_64 1/5 2023-01-11T21:10:56.8035302Z Verifying : nvidia-container-toolkit-base-1.11.0-1.x86_64 2/5 2023-01-11T21:10:56.8126549Z Verifying : nvidia-container-toolkit-1.11.0-1.x86_64 3/5 2023-01-11T21:10:56.8239188Z Verifying : libnvidia-container-tools-1.11.0-1.x86_64 4/5 2023-01-11T21:10:56.8997904Z Verifying : nvidia-docker2-2.11.0-1.noarch 5/5 2023-01-11T21:10:56.8998197Z 2023-01-11T21:10:56.8998308Z Installed: 2023-01-11T21:10:56.8998942Z nvidia-docker2.noarch 0:2.11.0-1 2023-01-11T21:10:56.8999155Z 2023-01-11T21:10:56.8999286Z Dependency Installed: 2023-01-11T21:10:56.8999729Z libnvidia-container-tools.x86_64 0:1.11.0-1 2023-01-11T21:10:56.9000198Z libnvidia-container1.x86_64 0:1.11.0-1 2023-01-11T21:10:56.9000652Z nvidia-container-toolkit.x86_64 0:1.11.0-1 2023-01-11T21:10:56.9001142Z nvidia-container-toolkit-base.x86_64 0:1.11.0-1 2023-01-11T21:10:56.9001583Z 2023-01-11T21:10:56.9001704Z Complete! 2023-01-11T21:10:57.0011085Z + sudo systemctl restart docker 2023-01-11T21:11:05.3016796Z Command completed after 1 attempt(s). 2023-01-11T21:11:05.3078500Z ##[group]Run python3 -m pip install psutil==5.9.1 2023-01-11T21:11:05.3078908Z python3 -m pip install psutil==5.9.1 2023-01-11T21:11:05.3079216Z python3 -m pip install pynvml==11.4.1 2023-01-11T21:11:05.3079570Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2023-01-11T21:11:05.3079950Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2023-01-11T21:11:05.3093889Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-01-11T21:11:05.3094192Z env: 2023-01-11T21:11:05.3094436Z GIT_DEFAULT_BRANCH: master 2023-01-11T21:11:05.3094686Z GPU_FLAG: --gpus all 2023-01-11T21:11:05.3094938Z ##[endgroup] 2023-01-11T21:11:05.8415423Z Defaulting to user installation because normal site-packages is not writeable 2023-01-11T21:11:06.2237477Z Collecting psutil==5.9.1 2023-01-11T21:11:06.2475111Z Downloading psutil-5.9.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (281 kB) 2023-01-11T21:11:06.3214055Z Installing collected packages: psutil 2023-01-11T21:11:06.4865745Z Successfully installed psutil-5.9.1 2023-01-11T21:11:06.9588149Z Defaulting to user installation because normal site-packages is not writeable 2023-01-11T21:11:07.0580455Z Collecting pynvml==11.4.1 2023-01-11T21:11:07.0746696Z Downloading pynvml-11.4.1-py3-none-any.whl (46 kB) 2023-01-11T21:11:07.1252935Z Installing collected packages: pynvml 2023-01-11T21:11:07.1805412Z Successfully installed pynvml-11.4.1 2023-01-11T21:11:07.2286082Z Prepare all required actions 2023-01-11T21:11:07.2286449Z Getting action download info 2023-01-11T21:11:07.5519732Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:4a8bfae15cc25cc0785c1603ee87a9da8fd442ea) 2023-01-11T21:11:07.7704108Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2023-01-11T21:11:07.9181721Z ##[group]Run ./.github/actions/download-build-artifacts 2023-01-11T21:11:07.9182026Z with: 2023-01-11T21:11:07.9182289Z name: linux-bionic-cuda11.6-py3.10-gcc7 2023-01-11T21:11:07.9182571Z env: 2023-01-11T21:11:07.9182808Z GIT_DEFAULT_BRANCH: master 2023-01-11T21:11:07.9183054Z GPU_FLAG: --gpus all 2023-01-11T21:11:07.9183302Z ##[endgroup] 2023-01-11T21:11:07.9216821Z ##[group]Run seemethere/download-artifact-s3@v4 2023-01-11T21:11:07.9217115Z with: 2023-01-11T21:11:07.9217391Z name: linux-bionic-cuda11.6-py3.10-gcc7 2023-01-11T21:11:07.9217679Z s3-bucket: gha-artifacts 2023-01-11T21:11:07.9218006Z region: us-east-1 2023-01-11T21:11:07.9218222Z env: 2023-01-11T21:11:07.9218458Z GIT_DEFAULT_BRANCH: master 2023-01-11T21:11:07.9218726Z GPU_FLAG: --gpus all 2023-01-11T21:11:07.9218956Z ##[endgroup] 2023-01-11T21:11:08.4724645Z Found 1 objects with prefix pytorch/pytorch/3896099317/linux-bionic-cuda11.6-py3.10-gcc7/ 2023-01-11T21:11:08.4725543Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2023-01-11T21:11:14.8926723Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2023-01-11T21:11:14.8927048Z 2023-01-11T21:11:14.8950657Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/ 2023-01-11T21:11:14.8960909Z Artifact download has finished successfully 2023-01-11T21:11:14.9112820Z ##[group]Run unzip -o artifacts.zip 2023-01-11T21:11:14.9113126Z unzip -o artifacts.zip 2023-01-11T21:11:14.9126690Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-01-11T21:11:14.9126985Z env: 2023-01-11T21:11:14.9127207Z GIT_DEFAULT_BRANCH: master 2023-01-11T21:11:14.9127473Z GPU_FLAG: --gpus all 2023-01-11T21:11:14.9127723Z ##[endgroup] 2023-01-11T21:11:14.9213542Z Archive: artifacts.zip 2023-01-11T21:11:14.9215605Z creating: dist/ 2023-01-11T21:11:17.0148535Z inflating: dist/torch-2.0.0a0+git8419ddd-cp310-cp310-linux_x86_64.whl 2023-01-11T21:11:17.0149000Z creating: build/custom_test_artifacts/ 2023-01-11T21:11:17.0149405Z creating: build/custom_test_artifacts/custom-op-build/ 2023-01-11T21:11:17.0149845Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2023-01-11T21:11:17.0156757Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeOutput.log 2023-01-11T21:11:17.0157295Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/ 2023-01-11T21:11:17.0157854Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeSystem.cmake 2023-01-11T21:11:17.0158389Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/ 2023-01-11T21:11:17.0158934Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/tmp/ 2023-01-11T21:11:17.0161407Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c 2023-01-11T21:11:17.0162529Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/a.out 2023-01-11T21:11:17.0163095Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/ 2023-01-11T21:11:17.0163644Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/ 2023-01-11T21:11:17.0166487Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp 2023-01-11T21:11:17.0167772Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out 2023-01-11T21:11:17.0169203Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin 2023-01-11T21:11:17.0170679Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake 2023-01-11T21:11:17.0172025Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin 2023-01-11T21:11:17.0173195Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake 2023-01-11T21:11:17.0173779Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/ 2023-01-11T21:11:17.0174348Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/ 2023-01-11T21:11:17.0228852Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2023-01-11T21:11:17.0229552Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2023-01-11T21:11:17.0230276Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2023-01-11T21:11:17.0231231Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2023-01-11T21:11:17.0231947Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2023-01-11T21:11:17.0232617Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2023-01-11T21:11:17.0233304Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2023-01-11T21:11:17.0233989Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2023-01-11T21:11:17.0234918Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2023-01-11T21:11:17.0276963Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2023-01-11T21:11:17.0318348Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2023-01-11T21:11:17.0319445Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2023-01-11T21:11:17.0320133Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2023-01-11T21:11:17.0320955Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c 2023-01-11T21:11:17.0321750Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin 2023-01-11T21:11:17.0322915Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2023-01-11T21:11:17.0323905Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o 2023-01-11T21:11:17.0326050Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu 2023-01-11T21:11:17.0398867Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out 2023-01-11T21:11:17.0472078Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin 2023-01-11T21:11:17.0472717Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake 2023-01-11T21:11:17.0473270Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2023-01-11T21:11:17.0474181Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeError.log 2023-01-11T21:11:17.0474744Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2023-01-11T21:11:17.0475286Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2023-01-11T21:11:17.0475870Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2023-01-11T21:11:17.0476474Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2023-01-11T21:11:17.0477049Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2023-01-11T21:11:17.0477773Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2023-01-11T21:11:17.0478505Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2023-01-11T21:11:17.0479550Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2023-01-11T21:11:17.0480331Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2023-01-11T21:11:17.0481070Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2023-01-11T21:11:17.0481758Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2023-01-11T21:11:17.0503237Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2023-01-11T21:11:17.0620640Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2023-01-11T21:11:17.0621205Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2023-01-11T21:11:17.0621777Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2023-01-11T21:11:17.0622415Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2023-01-11T21:11:17.0623016Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2023-01-11T21:11:17.0623606Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2023-01-11T21:11:17.0624192Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2023-01-11T21:11:17.0625046Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2023-01-11T21:11:17.0625653Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2023-01-11T21:11:17.0626419Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2023-01-11T21:11:17.0627032Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2023-01-11T21:11:17.0648171Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2023-01-11T21:11:17.0735085Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2023-01-11T21:11:17.0735726Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2023-01-11T21:11:17.0736342Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2023-01-11T21:11:17.0736870Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2023-01-11T21:11:17.0737590Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2023-01-11T21:11:17.0739190Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2023-01-11T21:11:17.0739717Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2023-01-11T21:11:17.0742569Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2023-01-11T21:11:17.0743272Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2023-01-11T21:11:17.0744012Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2023-01-11T21:11:17.0838971Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2023-01-11T21:11:17.0903865Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2023-01-11T21:11:17.0904327Z creating: build/custom_test_artifacts/jit-hook-build/ 2023-01-11T21:11:17.0904763Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2023-01-11T21:11:17.0911483Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeOutput.log 2023-01-11T21:11:17.0912015Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/ 2023-01-11T21:11:17.0912552Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeSystem.cmake 2023-01-11T21:11:17.0913090Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/ 2023-01-11T21:11:17.0913633Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/tmp/ 2023-01-11T21:11:17.0915784Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c 2023-01-11T21:11:17.0916940Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/a.out 2023-01-11T21:11:17.0917469Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/ 2023-01-11T21:11:17.0918017Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/ 2023-01-11T21:11:17.0920547Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp 2023-01-11T21:11:17.0921707Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out 2023-01-11T21:11:17.0923125Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin 2023-01-11T21:11:17.0923984Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake 2023-01-11T21:11:17.0925977Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin 2023-01-11T21:11:17.0926808Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake 2023-01-11T21:11:17.0927389Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/ 2023-01-11T21:11:17.0927936Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/ 2023-01-11T21:11:17.0982219Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2023-01-11T21:11:17.0982917Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2023-01-11T21:11:17.0983632Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2023-01-11T21:11:17.0984350Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2023-01-11T21:11:17.0985043Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2023-01-11T21:11:17.0985727Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2023-01-11T21:11:17.0986566Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2023-01-11T21:11:17.0987246Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2023-01-11T21:11:17.0988261Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2023-01-11T21:11:17.1030437Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2023-01-11T21:11:17.1071750Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2023-01-11T21:11:17.1072866Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2023-01-11T21:11:17.1073713Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2023-01-11T21:11:17.1074483Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c 2023-01-11T21:11:17.1075243Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin 2023-01-11T21:11:17.1076228Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2023-01-11T21:11:17.1077198Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o 2023-01-11T21:11:17.1079192Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu 2023-01-11T21:11:17.1152112Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out 2023-01-11T21:11:17.1225307Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin 2023-01-11T21:11:17.1225938Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake 2023-01-11T21:11:17.1226478Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2023-01-11T21:11:17.1227152Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeError.log 2023-01-11T21:11:17.1227676Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2023-01-11T21:11:17.1228205Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2023-01-11T21:11:17.1228790Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2023-01-11T21:11:17.1229405Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2023-01-11T21:11:17.1229986Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2023-01-11T21:11:17.1230558Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2023-01-11T21:11:17.1231323Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2023-01-11T21:11:17.1232371Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2023-01-11T21:11:17.1233133Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2023-01-11T21:11:17.1233720Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2023-01-11T21:11:17.1234305Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2023-01-11T21:11:17.1255538Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2023-01-11T21:11:17.1322303Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2023-01-11T21:11:17.1322928Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2023-01-11T21:11:17.1323526Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2023-01-11T21:11:17.1324056Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2023-01-11T21:11:17.1325182Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2023-01-11T21:11:17.1326308Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2023-01-11T21:11:17.1326973Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2023-01-11T21:11:17.1329762Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2023-01-11T21:11:17.1330721Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2023-01-11T21:11:17.1331929Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2023-01-11T21:11:17.1383152Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2023-01-11T21:11:17.1383631Z creating: build/custom_test_artifacts/custom-backend-build/ 2023-01-11T21:11:17.1384116Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2023-01-11T21:11:17.1390762Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeOutput.log 2023-01-11T21:11:17.1391333Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/ 2023-01-11T21:11:17.1391898Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeSystem.cmake 2023-01-11T21:11:17.1392477Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/ 2023-01-11T21:11:17.1393029Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/tmp/ 2023-01-11T21:11:17.1394662Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c 2023-01-11T21:11:17.1396342Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/a.out 2023-01-11T21:11:17.1396925Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/ 2023-01-11T21:11:17.1397481Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/ 2023-01-11T21:11:17.1399939Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp 2023-01-11T21:11:17.1401019Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out 2023-01-11T21:11:17.1403019Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin 2023-01-11T21:11:17.1403648Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake 2023-01-11T21:11:17.1405045Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin 2023-01-11T21:11:17.1406232Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake 2023-01-11T21:11:17.1406834Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/ 2023-01-11T21:11:17.1407398Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/ 2023-01-11T21:11:17.1461933Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2023-01-11T21:11:17.1462672Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2023-01-11T21:11:17.1463414Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2023-01-11T21:11:17.1464172Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2023-01-11T21:11:17.1464886Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2023-01-11T21:11:17.1465596Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2023-01-11T21:11:17.1466425Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2023-01-11T21:11:17.1467162Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2023-01-11T21:11:17.1467860Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2023-01-11T21:11:17.1509860Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2023-01-11T21:11:17.1550912Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2023-01-11T21:11:17.1551997Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2023-01-11T21:11:17.1552904Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2023-01-11T21:11:17.1553539Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c 2023-01-11T21:11:17.1554357Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin 2023-01-11T21:11:17.1555145Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2023-01-11T21:11:17.1556388Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o 2023-01-11T21:11:17.1558482Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu 2023-01-11T21:11:17.1631210Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out 2023-01-11T21:11:17.1704263Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin 2023-01-11T21:11:17.1704911Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake 2023-01-11T21:11:17.1705483Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2023-01-11T21:11:17.1706182Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeError.log 2023-01-11T21:11:17.1706781Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2023-01-11T21:11:17.1707346Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2023-01-11T21:11:17.1707958Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2023-01-11T21:11:17.1708569Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2023-01-11T21:11:17.1709192Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2023-01-11T21:11:17.1709804Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2023-01-11T21:11:17.1710799Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2023-01-11T21:11:17.1711755Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2023-01-11T21:11:17.1712508Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2023-01-11T21:11:17.1713120Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2023-01-11T21:11:17.1713744Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2023-01-11T21:11:17.1719208Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2023-01-11T21:11:17.1873171Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2023-01-11T21:11:17.1873838Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2023-01-11T21:11:17.1874483Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2023-01-11T21:11:17.1875149Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2023-01-11T21:11:17.1875773Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2023-01-11T21:11:17.1876394Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2023-01-11T21:11:17.1877040Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2023-01-11T21:11:17.1877680Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2023-01-11T21:11:17.1878297Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2023-01-11T21:11:17.1878924Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2023-01-11T21:11:17.1879536Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2023-01-11T21:11:17.1901027Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2023-01-11T21:11:17.1963458Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2023-01-11T21:11:17.1964108Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2023-01-11T21:11:17.1964742Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2023-01-11T21:11:17.1965327Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2023-01-11T21:11:17.1966068Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2023-01-11T21:11:17.1967152Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2023-01-11T21:11:17.1967918Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2023-01-11T21:11:17.1970666Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2023-01-11T21:11:17.1971822Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2023-01-11T21:11:17.1972427Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2023-01-11T21:11:17.2095473Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2023-01-11T21:11:17.2143109Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2023-01-11T21:11:17.2143469Z creating: build/lib/ 2023-01-11T21:11:17.2144104Z inflating: build/lib/libclog.a 2023-01-11T21:11:17.2214194Z inflating: build/lib/libgtest.a 2023-01-11T21:11:17.2224276Z inflating: build/lib/libpthreadpool.a 2023-01-11T21:11:17.2330439Z inflating: build/lib/libprotobuf-lite.a 2023-01-11T21:11:17.2426917Z inflating: build/lib/libbenchmark.a 2023-01-11T21:11:17.2436045Z inflating: build/lib/libittnotify.a 2023-01-11T21:11:17.2512775Z inflating: build/lib/libasmjit.a 2023-01-11T21:11:17.2544686Z inflating: build/lib/libtensorpipe_uv.a 2023-01-11T21:11:17.3077667Z inflating: build/lib/libprotobuf.a 2023-01-11T21:11:17.3218442Z inflating: build/lib/libgloo.a 2023-01-11T21:11:17.3250836Z inflating: build/lib/libfmt.a 2023-01-11T21:11:17.3252901Z inflating: build/lib/libcaffe2_nvrtc.so 2023-01-11T21:11:17.3253579Z inflating: build/lib/libfoxi_loader.a 2023-01-11T21:11:17.3336349Z inflating: build/lib/libc10.so 2023-01-11T21:11:17.3337648Z inflating: build/lib/libtorch_global_deps.so 2023-01-11T21:11:17.3347620Z inflating: build/lib/libcpuinfo.a 2023-01-11T21:11:17.3356501Z inflating: build/lib/libcpuinfo_internals.a 2023-01-11T21:11:17.3926231Z inflating: build/lib/libprotoc.a 2023-01-11T21:11:17.3942212Z inflating: build/lib/libqnnpack.a 2023-01-11T21:11:17.3965038Z inflating: build/lib/libpytorch_qnnpack.a 2023-01-11T21:11:17.3967615Z inflating: build/lib/libnnpack_reference_layers.a 2023-01-11T21:11:17.3989962Z inflating: build/lib/libnnpack.a 2023-01-11T21:11:17.4007593Z inflating: build/lib/libgmock.a 2023-01-11T21:11:17.4008244Z inflating: build/lib/libgtest_main.a 2023-01-11T21:11:17.4009264Z inflating: build/lib/libbenchmark_main.a 2023-01-11T21:11:17.4150914Z inflating: build/lib/libXNNPACK.a 2023-01-11T21:11:18.3920031Z inflating: build/lib/libdnnl.a 2023-01-11T21:11:18.4575908Z inflating: build/lib/libtensorpipe.a 2023-01-11T21:11:18.4630638Z inflating: build/lib/libc10_cuda.so 2023-01-11T21:11:18.6172985Z inflating: build/lib/libfbgemm.a 2023-01-11T21:11:18.6173531Z inflating: build/lib/libgmock_main.a 2023-01-11T21:11:18.7329099Z inflating: build/lib/libdnnl_graph.a 2023-01-11T21:11:18.7844830Z inflating: build/lib/libkineto.a 2023-01-11T21:11:18.8135101Z inflating: build/lib/libtensorpipe_cuda.a 2023-01-11T21:11:18.8180636Z inflating: build/lib/libcaffe2_protos.a 2023-01-11T21:11:18.8229172Z inflating: build/lib/libonnx_proto.a 2023-01-11T21:11:18.8907287Z inflating: build/lib/libonnx.a 2023-01-11T21:11:18.9339211Z inflating: build/lib/libgloo_cuda.a 2023-01-11T21:11:21.3140706Z inflating: build/lib/libtorch_cpu.so 2023-01-11T21:11:21.3151327Z inflating: build/lib/libunbox_lib.a 2023-01-11T21:11:23.4532830Z inflating: build/lib/libtorch_cuda.so 2023-01-11T21:11:23.4533626Z inflating: build/lib/libtorch.so 2023-01-11T21:11:23.4536837Z inflating: build/lib/libc10d_cuda_test.so 2023-01-11T21:11:24.4412506Z inflating: build/lib/libtorch_cuda_linalg.so 2023-01-11T21:11:24.4436625Z inflating: build/lib/libjitbackend_test.so 2023-01-11T21:11:24.4467718Z inflating: build/lib/libbackend_with_compiler.so 2023-01-11T21:11:24.4528825Z inflating: build/lib/libtorchbind_test.so 2023-01-11T21:11:24.4533990Z inflating: build/lib/libshm.so 2023-01-11T21:11:24.6386963Z inflating: build/lib/libtorch_python.so 2023-01-11T21:11:24.6427028Z inflating: build/lib/libnnapi_backend.so 2023-01-11T21:11:24.6427316Z creating: build/bin/ 2023-01-11T21:11:24.6481645Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2023-01-11T21:11:24.6539342Z inflating: build/bin/c10_DeviceGuard_test 2023-01-11T21:11:24.6595259Z inflating: build/bin/c10_Device_test 2023-01-11T21:11:24.6659967Z inflating: build/bin/c10_DispatchKeySet_test 2023-01-11T21:11:24.6713113Z inflating: build/bin/c10_StreamGuard_test 2023-01-11T21:11:24.6767736Z inflating: build/bin/c10_SymInt_test 2023-01-11T21:11:24.6829406Z inflating: build/bin/c10_InlineDeviceGuard_test 2023-01-11T21:11:24.6890954Z inflating: build/bin/c10_InlineStreamGuard_test 2023-01-11T21:11:24.6954077Z inflating: build/bin/c10_SizesAndStrides_test 2023-01-11T21:11:24.7007131Z inflating: build/bin/c10_Array_test 2023-01-11T21:11:24.7066154Z inflating: build/bin/c10_Bitset_test 2023-01-11T21:11:24.7149349Z inflating: build/bin/c10_C++17_test 2023-01-11T21:11:24.7202660Z inflating: build/bin/c10_ConstexprCrc_test 2023-01-11T21:11:24.7257176Z inflating: build/bin/c10_DeadlockDetection_test 2023-01-11T21:11:24.7312028Z inflating: build/bin/c10_Half_test 2023-01-11T21:11:24.7374863Z inflating: build/bin/c10_LeftRight_test 2023-01-11T21:11:24.7443836Z inflating: build/bin/c10_Metaprogramming_test 2023-01-11T21:11:24.7604146Z inflating: build/bin/c10_SmallVectorTest 2023-01-11T21:11:24.7659714Z inflating: build/bin/c10_Synchronized_test 2023-01-11T21:11:24.7722485Z inflating: build/bin/c10_ThreadLocal_test 2023-01-11T21:11:24.7781214Z inflating: build/bin/c10_TypeIndex_test 2023-01-11T21:11:24.7837002Z inflating: build/bin/c10_TypeList_test 2023-01-11T21:11:24.7890185Z inflating: build/bin/c10_TypeTraits_test 2023-01-11T21:11:24.7948145Z inflating: build/bin/c10_accumulate_test 2023-01-11T21:11:24.8010227Z inflating: build/bin/c10_bfloat16_test 2023-01-11T21:11:24.8071591Z inflating: build/bin/c10_complex_math_test 2023-01-11T21:11:24.8132469Z inflating: build/bin/c10_complex_test 2023-01-11T21:11:24.8252177Z inflating: build/bin/c10_either_test 2023-01-11T21:11:24.8310653Z inflating: build/bin/c10_exception_test 2023-01-11T21:11:24.8365794Z inflating: build/bin/c10_flags_test 2023-01-11T21:11:24.8550176Z inflating: build/bin/c10_intrusive_ptr_test 2023-01-11T21:11:24.8606113Z inflating: build/bin/c10_irange_test 2023-01-11T21:11:24.8669264Z inflating: build/bin/c10_logging_test 2023-01-11T21:11:24.8750647Z inflating: build/bin/c10_optional_test 2023-01-11T21:11:24.8820457Z inflating: build/bin/c10_ordered_preserving_dict_test 2023-01-11T21:11:24.8880992Z inflating: build/bin/c10_registry_test 2023-01-11T21:11:24.8945171Z inflating: build/bin/c10_string_view_test 2023-01-11T21:11:24.9002330Z inflating: build/bin/c10_tempfile_test 2023-01-11T21:11:24.9064226Z inflating: build/bin/c10_typeid_test 2023-01-11T21:11:24.9125128Z inflating: build/bin/c10_intrusive_ptr_benchmark 2023-01-11T21:11:24.9645968Z inflating: build/bin/protoc-3.13.0.0 2023-01-11T21:11:25.0166793Z inflating: build/bin/protoc 2023-01-11T21:11:25.0225776Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2023-01-11T21:11:25.0284744Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2023-01-11T21:11:25.0343764Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2023-01-11T21:11:25.0401497Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2023-01-11T21:11:25.0460577Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2023-01-11T21:11:25.0514090Z inflating: build/bin/c10_cuda_CUDATest 2023-01-11T21:11:25.0573182Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2023-01-11T21:11:25.0631697Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2023-01-11T21:11:25.0956156Z inflating: build/bin/vec_test_all_types_DEFAULT 2023-01-11T21:11:25.1315844Z inflating: build/bin/vec_test_all_types_AVX2 2023-01-11T21:11:25.1375450Z inflating: build/bin/FileStoreTest 2023-01-11T21:11:25.1441376Z inflating: build/bin/TCPStoreTest 2023-01-11T21:11:25.1501152Z inflating: build/bin/HashStoreTest 2023-01-11T21:11:25.1517259Z inflating: build/bin/ProcessGroupMPITest 2023-01-11T21:11:25.1580403Z inflating: build/bin/test_edge_op_registration 2023-01-11T21:11:25.1583704Z inflating: build/bin/example_allreduce 2023-01-11T21:11:25.1642432Z inflating: build/bin/Dimname_test 2023-01-11T21:11:25.1722673Z inflating: build/bin/Dict_test 2023-01-11T21:11:25.1793552Z inflating: build/bin/MaybeOwned_test 2023-01-11T21:11:25.1856489Z inflating: build/bin/NamedTensor_test 2023-01-11T21:11:25.1922061Z inflating: build/bin/apply_utils_test 2023-01-11T21:11:25.1986972Z inflating: build/bin/atest 2023-01-11T21:11:25.2054027Z inflating: build/bin/basic 2023-01-11T21:11:25.2113851Z inflating: build/bin/broadcast_test 2023-01-11T21:11:25.2177837Z inflating: build/bin/cpu_generator_test 2023-01-11T21:11:25.2233082Z inflating: build/bin/dispatch_key_set_test 2023-01-11T21:11:25.2291130Z inflating: build/bin/cpu_profiling_allocator_test 2023-01-11T21:11:25.2345954Z inflating: build/bin/dlconvertor_test 2023-01-11T21:11:25.2410192Z inflating: build/bin/extension_backend_test 2023-01-11T21:11:25.2471637Z inflating: build/bin/half_test 2023-01-11T21:11:25.2526254Z inflating: build/bin/lazy_tensor_test 2023-01-11T21:11:25.2586184Z inflating: build/bin/math_kernel_test 2023-01-11T21:11:25.2645932Z inflating: build/bin/memory_format_test 2023-01-11T21:11:25.2705482Z inflating: build/bin/memory_overlapping_test 2023-01-11T21:11:25.2761717Z inflating: build/bin/operator_name_test 2023-01-11T21:11:25.2825390Z inflating: build/bin/native_test 2023-01-11T21:11:25.2881094Z inflating: build/bin/operators_test 2023-01-11T21:11:25.2939762Z inflating: build/bin/packedtensoraccessor_test 2023-01-11T21:11:25.3003511Z inflating: build/bin/quantized_test 2023-01-11T21:11:25.3060101Z inflating: build/bin/reportMemoryUsage_test 2023-01-11T21:11:25.3122441Z inflating: build/bin/scalar_tensor_test 2023-01-11T21:11:25.3185640Z inflating: build/bin/scalar_test 2023-01-11T21:11:25.3243860Z inflating: build/bin/stride_properties_test 2023-01-11T21:11:25.3329911Z inflating: build/bin/tensor_iterator_test 2023-01-11T21:11:25.3385818Z inflating: build/bin/reduce_ops_test 2023-01-11T21:11:25.3458753Z inflating: build/bin/pow_test 2023-01-11T21:11:25.3462075Z inflating: build/bin/thread_init_test 2023-01-11T21:11:25.3525180Z inflating: build/bin/test_parallel 2023-01-11T21:11:25.3586349Z inflating: build/bin/type_ptr_test 2023-01-11T21:11:25.3641058Z inflating: build/bin/variant_test 2023-01-11T21:11:25.3699769Z inflating: build/bin/undefined_tensor_test 2023-01-11T21:11:25.3757708Z inflating: build/bin/mobile_memory_cleanup 2023-01-11T21:11:25.3759070Z inflating: build/bin/verify_api_visibility 2023-01-11T21:11:25.3835366Z inflating: build/bin/legacy_vmap_test 2023-01-11T21:11:25.3892046Z inflating: build/bin/weakref_test 2023-01-11T21:11:25.3948399Z inflating: build/bin/wrapdim_test 2023-01-11T21:11:25.4002456Z inflating: build/bin/xla_tensor_test 2023-01-11T21:11:25.4068655Z inflating: build/bin/IListRef_test 2023-01-11T21:11:25.4188356Z inflating: build/bin/List_test 2023-01-11T21:11:25.4260182Z inflating: build/bin/KernelFunction_test 2023-01-11T21:11:25.4393631Z inflating: build/bin/kernel_function_legacy_test 2023-01-11T21:11:25.4499245Z inflating: build/bin/kernel_function_test 2023-01-11T21:11:25.4596060Z inflating: build/bin/cpu_rng_test 2023-01-11T21:11:25.4736984Z inflating: build/bin/kernel_lambda_legacy_test 2023-01-11T21:11:25.4841933Z inflating: build/bin/ivalue_test 2023-01-11T21:11:25.4955529Z inflating: build/bin/kernel_lambda_test 2023-01-11T21:11:25.5022217Z inflating: build/bin/kernel_stackbased_test 2023-01-11T21:11:25.5078740Z inflating: build/bin/CppSignature_test 2023-01-11T21:11:25.5183795Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2023-01-11T21:11:25.5236757Z inflating: build/bin/op_allowlist_test 2023-01-11T21:11:25.5304058Z inflating: build/bin/type_test 2023-01-11T21:11:25.5366263Z inflating: build/bin/backend_fallback_test 2023-01-11T21:11:25.5680988Z inflating: build/bin/op_registration_test 2023-01-11T21:11:25.5740975Z inflating: build/bin/inline_container_test 2023-01-11T21:11:25.5799163Z inflating: build/bin/cuda_apply_test 2023-01-11T21:11:25.5865495Z inflating: build/bin/cuda_atomic_ops_test 2023-01-11T21:11:25.5925095Z inflating: build/bin/cuda_caching_host_allocator_test 2023-01-11T21:11:25.6003284Z inflating: build/bin/cuda_complex_math_test 2023-01-11T21:11:25.6067776Z inflating: build/bin/cuda_complex_test 2023-01-11T21:11:25.6122268Z inflating: build/bin/cuda_device_test 2023-01-11T21:11:25.6187621Z inflating: build/bin/cuda_cub_test 2023-01-11T21:11:25.6242988Z inflating: build/bin/cuda_dlconvertor_test 2023-01-11T21:11:25.6317226Z inflating: build/bin/cuda_distributions_test 2023-01-11T21:11:25.6373065Z inflating: build/bin/cuda_integer_divider_test 2023-01-11T21:11:25.6438029Z inflating: build/bin/cuda_generator_test 2023-01-11T21:11:25.6492578Z inflating: build/bin/cuda_half_test 2023-01-11T21:11:25.6559576Z inflating: build/bin/cuda_stream_test 2023-01-11T21:11:25.6618234Z inflating: build/bin/cuda_reportMemoryUsage_test 2023-01-11T21:11:25.6672115Z inflating: build/bin/cuda_optional_test 2023-01-11T21:11:25.6729030Z inflating: build/bin/cuda_packedtensoraccessor_test 2023-01-11T21:11:25.6784303Z inflating: build/bin/cuda_cudnn_test 2023-01-11T21:11:25.6842179Z inflating: build/bin/cuda_vectorized_test 2023-01-11T21:11:25.6860470Z inflating: build/bin/tutorial_tensorexpr 2023-01-11T21:11:25.6932515Z inflating: build/bin/ProcessGroupGlooTest 2023-01-11T21:11:25.6996861Z inflating: build/bin/ProcessGroupGlooAsyncTest 2023-01-11T21:11:25.7064879Z inflating: build/bin/ProcessGroupNCCLTest 2023-01-11T21:11:25.7128948Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2023-01-11T21:11:25.7187810Z inflating: build/bin/ProcessGroupUCCTest 2023-01-11T21:11:25.7247578Z inflating: build/bin/test_dist_autograd 2023-01-11T21:11:25.7324375Z inflating: build/bin/test_cpp_rpc 2023-01-11T21:11:25.7327050Z inflating: build/bin/parallel_benchmark 2023-01-11T21:11:25.7402682Z inflating: build/bin/test_mobile_nnc 2023-01-11T21:11:25.7414422Z inflating: build/bin/aot_model_compiler_test 2023-01-11T21:11:25.8322242Z inflating: build/bin/test_tensorexpr 2023-01-11T21:11:25.8328349Z inflating: build/bin/torch_shm_manager 2023-01-11T21:11:25.8717287Z inflating: build/bin/test_lazy 2023-01-11T21:11:26.0037348Z inflating: build/bin/test_api 2023-01-11T21:11:26.1241580Z inflating: build/bin/test_jit 2023-01-11T21:11:26.1243614Z inflating: .pytorch-test-times.json 2023-01-11T21:11:26.1275469Z ##[group]Run df -H 2023-01-11T21:11:26.1275707Z df -H 2023-01-11T21:11:26.1289378Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-01-11T21:11:26.1289654Z env: 2023-01-11T21:11:26.1289894Z GIT_DEFAULT_BRANCH: master 2023-01-11T21:11:26.1290159Z GPU_FLAG: --gpus all 2023-01-11T21:11:26.1290786Z ##[endgroup] 2023-01-11T21:11:26.1330536Z Filesystem Size Used Avail Use% Mounted on 2023-01-11T21:11:26.1330896Z devtmpfs 129G 0 129G 0% /dev 2023-01-11T21:11:26.1331179Z tmpfs 129G 0 129G 0% /dev/shm 2023-01-11T21:11:26.1331436Z tmpfs 129G 607k 129G 1% /run 2023-01-11T21:11:26.1331723Z tmpfs 129G 0 129G 0% /sys/fs/cgroup 2023-01-11T21:11:26.1332016Z /dev/xvda1 162G 29G 133G 18% / 2023-01-11T21:11:26.1332515Z tmpfs 26G 0 26G 0% /run/user/0 2023-01-11T21:11:26.1357513Z ##[group]Run .github/scripts/parse_ref.py 2023-01-11T21:11:26.1357886Z .github/scripts/parse_ref.py 2023-01-11T21:11:26.1370774Z shell: /usr/bin/bash -e {0} 2023-01-11T21:11:26.1371027Z env: 2023-01-11T21:11:26.1371263Z GIT_DEFAULT_BRANCH: master 2023-01-11T21:11:26.1371511Z GPU_FLAG: --gpus all 2023-01-11T21:11:26.1371760Z ##[endgroup] 2023-01-11T21:11:26.1672814Z ##[group]Run set -x 2023-01-11T21:11:26.1673188Z set -x 2023-01-11T21:11:26.1673415Z  2023-01-11T21:11:26.1673669Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2023-01-11T21:11:26.1674017Z  TEST_COMMAND=.jenkins/pytorch/multigpu-test.sh 2023-01-11T21:11:26.1674363Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2023-01-11T21:11:26.1674680Z  TEST_COMMAND=.jenkins/onnx/test.sh 2023-01-11T21:11:26.1674940Z else 2023-01-11T21:11:26.1675218Z  TEST_COMMAND=.jenkins/pytorch/test.sh 2023-01-11T21:11:26.1675488Z fi 2023-01-11T21:11:26.1675688Z  2023-01-11T21:11:26.1676006Z COMMIT_MESSAGES=$(git cherry -v "origin/${GIT_DEFAULT_BRANCH:-master}") 2023-01-11T21:11:26.1676320Z  2023-01-11T21:11:26.1676596Z # sanitize the input commit message and PR body here: 2023-01-11T21:11:26.1676884Z # 2023-01-11T21:11:26.1677264Z # trim all new lines from commit messages + PR_BODY to avoid issues with batch environment 2023-01-11T21:11:26.1677755Z # variable copying. see https://github.com/pytorch/pytorch/pull/80043#issuecomment-1167796028 2023-01-11T21:11:26.1678176Z COMMIT_MESSAGES="${COMMIT_MESSAGES//[$'\n\r']}" 2023-01-11T21:11:26.1678488Z PR_BODY="${PR_BODY//[$'\n\r']}" 2023-01-11T21:11:26.1678743Z  2023-01-11T21:11:26.1679081Z # then trim all special characters like single and double quotes to avoid unescaped inputs to 2023-01-11T21:11:26.1679457Z # wreak havoc internally 2023-01-11T21:11:26.1679779Z export COMMIT_MESSAGES="${COMMIT_MESSAGES//[\'\"]}" 2023-01-11T21:11:26.1680090Z export PR_BODY="${PR_BODY//[\'\"]}" 2023-01-11T21:11:26.1680346Z  2023-01-11T21:11:26.1680654Z # detached container should get cleaned up by teardown_ec2_linux 2023-01-11T21:11:26.1681036Z # TODO: Stop building test binaries as part of the build phase 2023-01-11T21:11:26.1681408Z # Used for GPU_FLAG since that doesn't play nice 2023-01-11T21:11:26.1681734Z # shellcheck disable=SC2086,SC2090 2023-01-11T21:11:26.1682033Z container_name=$(docker run \ 2023-01-11T21:11:26.1682292Z  ${GPU_FLAG:-} \ 2023-01-11T21:11:26.1682564Z  -e BUILD_ENVIRONMENT \ 2023-01-11T21:11:26.1682836Z  -e PR_NUMBER \ 2023-01-11T21:11:26.1683083Z  -e GITHUB_ACTIONS \ 2023-01-11T21:11:26.1683472Z  -e BASE_SHA \ 2023-01-11T21:11:26.1683721Z  -e BRANCH \ 2023-01-11T21:11:26.1683944Z  -e SHA1 \ 2023-01-11T21:11:26.1684200Z  -e AWS_DEFAULT_REGION \ 2023-01-11T21:11:26.1684473Z  -e IN_WHEEL_TEST \ 2023-01-11T21:11:26.1684716Z  -e SHARD_NUMBER \ 2023-01-11T21:11:26.1684977Z  -e TEST_CONFIG \ 2023-01-11T21:11:26.1685238Z  -e NUM_TEST_SHARDS \ 2023-01-11T21:11:26.1685492Z  -e PR_BODY \ 2023-01-11T21:11:26.1685739Z  -e COMMIT_MESSAGES \ 2023-01-11T21:11:26.1686021Z  -e CONTINUE_THROUGH_ERROR \ 2023-01-11T21:11:26.1686320Z  -e PYTORCH_RETRY_TEST_CASES \ 2023-01-11T21:11:26.1686614Z  -e PYTORCH_OVERRIDE_FLAKY_SIGNAL \ 2023-01-11T21:11:26.1686902Z  -e PR_LABELS \ 2023-01-11T21:11:26.1687188Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2023-01-11T21:11:26.1687456Z  -e SCCACHE_BUCKET \ 2023-01-11T21:11:26.1687730Z  -e SCCACHE_S3_KEY_PREFIX \ 2023-01-11T21:11:26.1688001Z  -e XLA_CUDA \ 2023-01-11T21:11:26.1688266Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2023-01-11T21:11:26.1688584Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2023-01-11T21:11:26.1688914Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2023-01-11T21:11:26.1689260Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2023-01-11T21:11:26.1689564Z  --ulimit stack=10485760:83886080 \ 2023-01-11T21:11:26.1689938Z  --security-opt seccomp=unconfined \ 2023-01-11T21:11:26.1690250Z  --cap-add=SYS_PTRACE \ 2023-01-11T21:11:26.1690746Z  --ipc=host \ 2023-01-11T21:11:26.1691016Z  --shm-size="${SHM_SIZE}" \ 2023-01-11T21:11:26.1691275Z  --tty \ 2023-01-11T21:11:26.1691497Z  --detach \ 2023-01-11T21:11:26.1691767Z  --name="${container_name}" \ 2023-01-11T21:11:26.1692041Z  --user jenkins \ 2023-01-11T21:11:26.1692348Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2023-01-11T21:11:26.1692687Z  -w /var/lib/jenkins/workspace \ 2023-01-11T21:11:26.1692967Z  "${DOCKER_IMAGE}" 2023-01-11T21:11:26.1693204Z ) 2023-01-11T21:11:26.1693487Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2023-01-11T21:11:26.1693971Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2023-01-11T21:11:26.1706155Z shell: /usr/bin/bash -e {0} 2023-01-11T21:11:26.1706385Z env: 2023-01-11T21:11:26.1706621Z GIT_DEFAULT_BRANCH: master 2023-01-11T21:11:26.1706883Z GPU_FLAG: --gpus all 2023-01-11T21:11:26.1707194Z BUILD_ENVIRONMENT: linux-bionic-cuda11.6-py3.10-gcc7 2023-01-11T21:11:26.1707507Z PR_NUMBER: 91627 2023-01-11T21:11:26.1707751Z BRANCH: pull/91627 2023-01-11T21:11:26.1708026Z SHA1: 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e 2023-01-11T21:11:26.1708357Z BASE_SHA: db2a237763eb8693a20788be94f8c192e762baa8 2023-01-11T21:11:26.1708661Z PYTORCH_RETRY_TEST_CASES: 1 2023-01-11T21:11:26.1708926Z PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1 2023-01-11T21:11:26.1709202Z TEST_CONFIG: distributed 2023-01-11T21:11:26.1709451Z SHARD_NUMBER: 1 2023-01-11T21:11:26.1709688Z NUM_TEST_SHARDS: 3 2023-01-11T21:11:26.1709950Z PR_BODY: Fixes #91003 cc @ezyang @gchanan 2023-01-11T21:11:26.1710242Z CONTINUE_THROUGH_ERROR: False 2023-01-11T21:11:26.1710573Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2023-01-11T21:11:26.1710885Z SCCACHE_S3_KEY_PREFIX: pull 2023-01-11T21:11:26.1711137Z SHM_SIZE: 2g 2023-01-11T21:11:26.1711626Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd 2023-01-11T21:11:26.1712079Z XLA_CUDA: 2023-01-11T21:11:26.1712425Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2023-01-11T21:11:26.1712803Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2023-01-11T21:11:26.1759404Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2023-01-11T21:11:26.1759735Z ##[endgroup] 2023-01-11T21:11:26.1788645Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2023-01-11T21:11:26.1789177Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *onnx* ]] 2023-01-11T21:11:26.1789514Z + TEST_COMMAND=.jenkins/pytorch/test.sh 2023-01-11T21:11:26.1792699Z ++ git cherry -v origin/master 2023-01-11T21:11:26.2331540Z + COMMIT_MESSAGES='+ 52a16ce42647731c772e14e7175afa40fda07b3d make torchgen rename also Number arguments into '\''input'\'' 2023-01-11T21:11:26.2332009Z + 87db01a53ecb702267ec36787654e418a52f8e93 fix torch.where signature mismatch 2023-01-11T21:11:26.2332562Z + 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e '\''other'\'' instead of '\''output'\'' in documentation' 2023-01-11T21:11:26.2333755Z + COMMIT_MESSAGES='+ 52a16ce42647731c772e14e7175afa40fda07b3d make torchgen rename also Number arguments into '\''input'\''+ 87db01a53ecb702267ec36787654e418a52f8e93 fix torch.where signature mismatch+ 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e '\''other'\'' instead of '\''output'\'' in documentation' 2023-01-11T21:11:26.2334438Z + PR_BODY='Fixes #91003 cc @ezyang @gchanan' 2023-01-11T21:11:26.2336551Z + export 'COMMIT_MESSAGES=+ 52a16ce42647731c772e14e7175afa40fda07b3d make torchgen rename also Number arguments into input+ 87db01a53ecb702267ec36787654e418a52f8e93 fix torch.where signature mismatch+ 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e other instead of output in documentation' 2023-01-11T21:11:26.2337836Z + COMMIT_MESSAGES='+ 52a16ce42647731c772e14e7175afa40fda07b3d make torchgen rename also Number arguments into input+ 87db01a53ecb702267ec36787654e418a52f8e93 fix torch.where signature mismatch+ 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e other instead of output in documentation' 2023-01-11T21:11:26.2338493Z + export 'PR_BODY=Fixes #91003 cc @ezyang @gchanan' 2023-01-11T21:11:26.2338846Z + PR_BODY='Fixes #91003 cc @ezyang @gchanan' 2023-01-11T21:11:26.2345577Z +++ nproc --ignore=2 2023-01-11T21:11:26.2378241Z ++ docker run --gpus all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e PR_BODY -e COMMIT_MESSAGES -e CONTINUE_THROUGH_ERROR -e PYTORCH_RETRY_TEST_CASES -e PYTORCH_OVERRIDE_FLAKY_SIGNAL -e PR_LABELS -e MAX_JOBS=30 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS --env-file=/tmp/github_env_3896099317 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd 2023-01-11T21:11:40.6236963Z + container_name=76a1849f93d45a34422360460369a49f22eebafa5f0784e84d188aa13b21145b 2023-01-11T21:11:40.6237530Z + echo DOCKER_CONTAINER_ID=76a1849f93d45a34422360460369a49f22eebafa5f0784e84d188aa13b21145b 2023-01-11T21:11:40.6242267Z ++ echo dist/torch-2.0.0a0+git8419ddd-cp310-cp310-linux_x86_64.whl 2023-01-11T21:11:40.6244349Z + docker exec -t 76a1849f93d45a34422360460369a49f22eebafa5f0784e84d188aa13b21145b sh -c 'pip install dist/torch-2.0.0a0+git8419ddd-cp310-cp310-linux_x86_64.whl[opt-einsum] && .jenkins/pytorch/test.sh' 2023-01-11T21:11:41.1672920Z Processing ./dist/torch-2.0.0a0+git8419ddd-cp310-cp310-linux_x86_64.whl 2023-01-11T21:11:42.1308451Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.10/site-packages (from torch==2.0.0a0+git8419ddd) (4.4.0) 2023-01-11T21:11:42.1311302Z Requirement already satisfied: networkx in /opt/conda/lib/python3.10/site-packages (from torch==2.0.0a0+git8419ddd) (2.6.3) 2023-01-11T21:11:42.1315527Z Requirement already satisfied: sympy in /opt/conda/lib/python3.10/site-packages (from torch==2.0.0a0+git8419ddd) (1.11.1) 2023-01-11T21:11:42.1331603Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/lib/python3.10/site-packages (from torch==2.0.0a0+git8419ddd) (3.3.0) 2023-01-11T21:11:42.1407861Z Requirement already satisfied: numpy>=1.7 in /opt/conda/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.0.0a0+git8419ddd) (1.21.2) 2023-01-11T21:11:42.1619224Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.10/site-packages (from sympy->torch==2.0.0a0+git8419ddd) (1.2.1) 2023-01-11T21:11:43.0519938Z Installing collected packages: torch 2023-01-11T21:11:52.5818091Z Successfully installed torch-2.0.0a0+git8419ddd 2023-01-11T21:11:52.7341102Z + echo 'Environment variables:' 2023-01-11T21:11:52.7341430Z Environment variables: 2023-01-11T21:11:52.7341708Z + env 2023-01-11T21:11:52.7349099Z SHARD_NUMBER=1 2023-01-11T21:11:52.7349535Z NV_LIBCUBLAS_DEV_VERSION=11.9.2.110-1 2023-01-11T21:11:52.7349921Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-6 2023-01-11T21:11:52.7350275Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2023-01-11T21:11:52.7353170Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.12.10-1+cuda11.6 2023-01-11T21:11:52.7353510Z UCC_HOME=/usr 2023-01-11T21:11:52.7353906Z BUILD_ENVIRONMENT=linux-bionic-cuda11.6-py3.10-gcc7 2023-01-11T21:11:52.7354260Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2023-01-11T21:11:52.7354678Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-6=11.6.3.124-1 2023-01-11T21:11:52.7355687Z INSTALLED_DB=yes 2023-01-11T21:11:52.7357482Z HOSTNAME=76a1849f93d4 2023-01-11T21:11:52.7358056Z GITHUB_REF_NAME=91627/merge 2023-01-11T21:11:52.7358381Z GITHUB_API_URL=https://api.github.com 2023-01-11T21:11:52.7358701Z GITHUB_REPOSITORY_OWNER_ID=21003710 2023-01-11T21:11:52.7358965Z OPENSSL_DIR=/opt/openssl 2023-01-11T21:11:52.7359282Z UCC_COMMIT=1c7a7127186e7836f73aafbd7697bbc274a77eee 2023-01-11T21:11:52.7359930Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_f3b45a2b-8801-4b5f-9961-1de1a8e6fa84 2023-01-11T21:11:52.7360327Z CUDA_PATH=/usr/local/cuda 2023-01-11T21:11:52.7360828Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2023-01-11T21:11:52.7361205Z GITHUB_RUN_ATTEMPT=2 2023-01-11T21:11:52.7361446Z TEST_CONFIG=distributed 2023-01-11T21:11:52.7361756Z NV_LIBNPP_VERSION=11.6.3.124-1 2023-01-11T21:11:52.7362134Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-6=11.6.124-1 2023-01-11T21:11:52.7362448Z GITHUB_REPOSITORY_OWNER=pytorch 2023-01-11T21:11:52.7362698Z GITHUB_ACTIONS=true 2023-01-11T21:11:52.7363030Z NVIDIA_VISIBLE_DEVICES=all 2023-01-11T21:11:52.7363339Z NV_NVPROF_VERSION=11.6.124-1 2023-01-11T21:11:52.7363642Z NV_LIBCUSPARSE_VERSION=11.7.2.124-1 2023-01-11T21:11:52.7364032Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/91627/merge 2023-01-11T21:11:52.7364397Z NVIDIA_PRODUCT_NAME=CUDA 2023-01-11T21:11:52.7364617Z CI=true 2023-01-11T21:11:52.7364870Z PYTORCH_OVERRIDE_FLAKY_SIGNAL=1 2023-01-11T21:11:52.7365269Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-6=11.9.2.110-1 2023-01-11T21:11:52.7365561Z BRANCH=pull/91627 2023-01-11T21:11:52.7365814Z GITHUB_HEAD_REF=master 2023-01-11T21:11:52.7366142Z UCX_COMMIT=31e74cac7bee0ef66bef2af72e7d86d9c282e5ab 2023-01-11T21:11:52.7366440Z GITHUB_ACTOR=LucaLumetti 2023-01-11T21:11:52.7366746Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2023-01-11T21:11:52.7367038Z GITHUB_ACTION_REF= 2023-01-11T21:11:52.7367319Z NCCL_VERSION=2.12.10-1 2023-01-11T21:11:52.7367558Z GITHUB_ACTION=__self 2023-01-11T21:11:52.7367818Z GITHUB_REF_PROTECTED=false 2023-01-11T21:11:52.7371621Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2023-01-11T21:11:52.7372502Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2023-01-11T21:11:52.7375989Z *** 2023-01-11T21:11:52.7376480Z INSTALLED_VISION=yes 2023-01-11T21:11:52.7377146Z NVARCH=x86_64 2023-01-11T21:11:52.7377811Z NV_LIBCUSPARSE_DEV_VERSION=11.7.2.124-1 2023-01-11T21:11:52.7378194Z HOME=/var/lib/jenkins 2023-01-11T21:11:52.7378740Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_f3b45a2b-8801-4b5f-9961-1de1a8e6fa84 2023-01-11T21:11:52.7379313Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2023-01-11T21:11:52.7379593Z GITHUB_ACTION_REPOSITORY= 2023-01-11T21:11:52.7379857Z GITHUB_REF_TYPE=branch 2023-01-11T21:11:52.7380152Z NV_LIBNCCL_PACKAGE_VERSION=2.12.10-1 2023-01-11T21:11:52.7380434Z GITHUB_RETENTION_DAYS=90 2023-01-11T21:11:52.7380810Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2023-01-11T21:11:52.7381198Z NV_LIBNCCL_PACKAGE=libnccl2=2.12.10-1+cuda11.6 2023-01-11T21:11:52.7381749Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_f3b45a2b-8801-4b5f-9961-1de1a8e6fa84 2023-01-11T21:11:52.7382150Z DEBIAN_FRONTEND=noninteractive 2023-01-11T21:11:52.7382497Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2023-01-11T21:11:52.7382784Z GITHUB_REF=refs/pull/91627/merge 2023-01-11T21:11:52.7383081Z NV_CUDA_LIB_VERSION=11.6.2-1 2023-01-11T21:11:52.7383385Z GITHUB_SHA=57fc38f02f250896a12b32cfa200a6105a03d09c 2023-01-11T21:11:52.7383661Z INSTALLED_PROTOBUF=yes 2023-01-11T21:11:52.7383937Z GITHUB_REPOSITORY_ID=65600975 2023-01-11T21:11:52.7384201Z GITHUB_RUN_ID=3896099317 2023-01-11T21:11:52.7384531Z NV_LIBNPP_PACKAGE=libnpp-11-6=11.6.3.124-1 2023-01-11T21:11:52.7384829Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2023-01-11T21:11:52.7385123Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2023-01-11T21:11:52.7385417Z NV_NVTX_VERSION=11.6.124-1 2023-01-11T21:11:52.7385684Z CONTINUE_THROUGH_ERROR=False 2023-01-11T21:11:52.7385989Z GITHUB_SERVER_URL=https://github.com 2023-01-11T21:11:52.7386336Z MAX_JOBS=30 2023-01-11T21:11:52.7386593Z GITHUB_ACTOR_ID=7543386 2023-01-11T21:11:52.7386892Z NV_LIBCUBLAS_VERSION=11.9.2.110-1 2023-01-11T21:11:52.7387246Z NV_LIBCUBLAS_PACKAGE=libcublas-11-6=11.9.2.110-1 2023-01-11T21:11:52.7387725Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2023-01-11T21:11:52.7388066Z UCX_HOME=/usr 2023-01-11T21:11:52.7388300Z PYTORCH_RETRY_TEST_CASES=1 2023-01-11T21:11:52.7388631Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2023-01-11T21:11:52.7388983Z BASE_SHA=db2a237763eb8693a20788be94f8c192e762baa8 2023-01-11T21:11:52.7389325Z NV_CUDA_CUDART_DEV_VERSION=11.6.55-1 2023-01-11T21:11:52.7389604Z PR_BODY=Fixes #91003 cc @ezyang @gchanan 2023-01-11T21:11:52.7389879Z GITHUB_BASE_REF=master 2023-01-11T21:11:52.7390119Z TERM=xterm 2023-01-11T21:11:52.7390319Z XLA_CUDA= 2023-01-11T21:11:52.7390587Z NV_NVML_DEV_VERSION=11.6.55-1 2023-01-11T21:11:52.7390857Z TORCH_CUDA_ARCH_LIST=Maxwell 2023-01-11T21:11:52.7391098Z CUDA_VERSION=11.6.2 2023-01-11T21:11:52.7391439Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-6 2023-01-11T21:11:52.7391738Z OPENSSL_ROOT_DIR=/opt/openssl 2023-01-11T21:11:52.7392260Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_f3b45a2b-8801-4b5f-9961-1de1a8e6fa84 2023-01-11T21:11:52.7392638Z GITHUB_JOB=test 2023-01-11T21:11:52.7392892Z SCCACHE_S3_KEY_PREFIX=pull 2023-01-11T21:11:52.7393469Z COMMIT_MESSAGES=+ 52a16ce42647731c772e14e7175afa40fda07b3d make torchgen rename also Number arguments into input+ 87db01a53ecb702267ec36787654e418a52f8e93 fix torch.where signature mismatch+ 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e other instead of output in documentation 2023-01-11T21:11:52.7394146Z NVIDIA_DRIVER_CAPABILITIES=compute,utility 2023-01-11T21:11:52.7394433Z NUM_TEST_SHARDS=3 2023-01-11T21:11:52.7394674Z PR_NUMBER=91627 2023-01-11T21:11:52.7395239Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_f3b45a2b-8801-4b5f-9961-1de1a8e6fa84 2023-01-11T21:11:52.7395611Z SHLVL=1 2023-01-11T21:11:52.7395955Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-6 2023-01-11T21:11:52.7396279Z GITHUB_REPOSITORY=pytorch/pytorch 2023-01-11T21:11:52.7397050Z NVIDIA_REQUIRE_CUDA=cuda>=11.6 brand=tesla,driver>=418,driver<419 brand=tesla,driver>=450,driver<451 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 2023-01-11T21:11:52.7397890Z NV_LIBNPP_DEV_VERSION=11.6.3.124-1 2023-01-11T21:11:52.7398204Z SHA1=8419ddda87c8a47eacc63b54bc7ec98c1f27c26e 2023-01-11T21:11:52.7398508Z GITHUB_EVENT_NAME=pull_request 2023-01-11T21:11:52.7398799Z NV_CUDA_CUDART_VERSION=11.6.55-1 2023-01-11T21:11:52.7399143Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2023-01-11T21:11:52.7399431Z GITHUB_RUN_NUMBER=77928 2023-01-11T21:11:52.7399676Z GITHUB_WORKFLOW=pull 2023-01-11T21:11:52.7400083Z PATH=/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2023-01-11T21:11:52.7400527Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.12.10-1 2023-01-11T21:11:52.7400845Z GITHUB_WORKFLOW_SHA=57fc38f02f250896a12b32cfa200a6105a03d09c 2023-01-11T21:11:52.7401296Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2023-01-11T21:11:52.7401642Z GITHUB_TRIGGERING_ACTOR=albanD 2023-01-11T21:11:52.7401883Z _=/usr/bin/env 2023-01-11T21:11:52.7402259Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2023-01-11T21:11:52.7570264Z + TORCH_INSTALL_DIR=/opt/conda/lib/python3.10/site-packages/torch 2023-01-11T21:11:52.7571272Z + TORCH_BIN_DIR=/opt/conda/lib/python3.10/site-packages/torch/bin 2023-01-11T21:11:52.7571709Z + TORCH_LIB_DIR=/opt/conda/lib/python3.10/site-packages/torch/lib 2023-01-11T21:11:52.7572336Z + TORCH_TEST_DIR=/opt/conda/lib/python3.10/site-packages/torch/test 2023-01-11T21:11:52.7572682Z + BUILD_DIR=build 2023-01-11T21:11:52.7572924Z + BUILD_RENAMED_DIR=build_renamed 2023-01-11T21:11:52.7573195Z + BUILD_BIN_DIR=build/bin 2023-01-11T21:11:52.7573462Z + export VALGRIND=ON 2023-01-11T21:11:52.7573688Z + VALGRIND=ON 2023-01-11T21:11:52.7573956Z + export TORCH_INDUCTOR_INSTALL_GXX=ON 2023-01-11T21:11:52.7574251Z + TORCH_INDUCTOR_INSTALL_GXX=ON 2023-01-11T21:11:52.7574666Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *clang9* ]] 2023-01-11T21:11:52.7575094Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 != *bazel* ]] 2023-01-11T21:11:52.7576242Z ++ realpath build/custom_test_artifacts 2023-01-11T21:11:52.7583537Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2023-01-11T21:11:52.7587272Z ++ dirname .jenkins/pytorch/test.sh 2023-01-11T21:11:52.7593677Z + source .jenkins/pytorch/common.sh 2023-01-11T21:11:52.7599348Z +++ dirname .jenkins/pytorch/common.sh 2023-01-11T21:11:52.7608608Z ++ source .jenkins/pytorch/common_utils.sh 2023-01-11T21:11:52.7610995Z +++ declare -f -t trap_add 2023-01-11T21:11:52.7617417Z ++ set -ex 2023-01-11T21:11:52.7617809Z ++ [[ linux-bionic-cuda11.6-py3.10-gcc7 == *rocm* ]] 2023-01-11T21:11:52.7618108Z ++ BUILD_TEST_LIBTORCH=0 2023-01-11T21:11:52.7618435Z + echo 'Environment variables' 2023-01-11T21:11:52.7618707Z Environment variables 2023-01-11T21:11:52.7618928Z + env 2023-01-11T21:11:52.7625035Z SHARD_NUMBER=1 2023-01-11T21:11:52.7625368Z NV_LIBCUBLAS_DEV_VERSION=11.9.2.110-1 2023-01-11T21:11:52.7625936Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-6 2023-01-11T21:11:52.7626473Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2023-01-11T21:11:52.7626907Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.12.10-1+cuda11.6 2023-01-11T21:11:52.7627194Z UCC_HOME=/usr 2023-01-11T21:11:52.7627707Z BUILD_ENVIRONMENT=linux-bionic-cuda11.6-py3.10-gcc7 2023-01-11T21:11:52.7628097Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2023-01-11T21:11:52.7628492Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-6=11.6.3.124-1 2023-01-11T21:11:52.7628798Z INSTALLED_DB=yes 2023-01-11T21:11:52.7629056Z HOSTNAME=76a1849f93d4 2023-01-11T21:11:52.7629315Z GITHUB_REF_NAME=91627/merge 2023-01-11T21:11:52.7629625Z GITHUB_API_URL=https://api.github.com 2023-01-11T21:11:52.7629926Z GITHUB_REPOSITORY_OWNER_ID=21003710 2023-01-11T21:11:52.7630206Z OPENSSL_DIR=/opt/openssl 2023-01-11T21:11:52.7630513Z UCC_COMMIT=1c7a7127186e7836f73aafbd7697bbc274a77eee 2023-01-11T21:11:52.7631096Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_f3b45a2b-8801-4b5f-9961-1de1a8e6fa84 2023-01-11T21:11:52.7631702Z CUDA_PATH=/usr/local/cuda 2023-01-11T21:11:52.7632197Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2023-01-11T21:11:52.7632550Z GITHUB_RUN_ATTEMPT=2 2023-01-11T21:11:52.7632809Z TEST_CONFIG=distributed 2023-01-11T21:11:52.7633106Z NV_LIBNPP_VERSION=11.6.3.124-1 2023-01-11T21:11:52.7633465Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-6=11.6.124-1 2023-01-11T21:11:52.7633788Z GITHUB_REPOSITORY_OWNER=pytorch 2023-01-11T21:11:52.7634062Z GITHUB_ACTIONS=true 2023-01-11T21:11:52.7634302Z NVIDIA_VISIBLE_DEVICES=all 2023-01-11T21:11:52.7634607Z NV_NVPROF_VERSION=11.6.124-1 2023-01-11T21:11:52.7634919Z NV_LIBCUSPARSE_VERSION=11.7.2.124-1 2023-01-11T21:11:52.7635279Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/91627/merge 2023-01-11T21:11:52.7635634Z NVIDIA_PRODUCT_NAME=CUDA 2023-01-11T21:11:52.7635880Z CI=true 2023-01-11T21:11:52.7636135Z PYTORCH_OVERRIDE_FLAKY_SIGNAL=1 2023-01-11T21:11:52.7636522Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-6=11.9.2.110-1 2023-01-11T21:11:52.7636827Z BRANCH=pull/91627 2023-01-11T21:11:52.7637080Z GITHUB_HEAD_REF=master 2023-01-11T21:11:52.7637376Z UCX_COMMIT=31e74cac7bee0ef66bef2af72e7d86d9c282e5ab 2023-01-11T21:11:52.7637691Z GITHUB_ACTOR=LucaLumetti 2023-01-11T21:11:52.7637999Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2023-01-11T21:11:52.7638274Z GITHUB_ACTION_REF= 2023-01-11T21:11:52.7638636Z NCCL_VERSION=2.12.10-1 2023-01-11T21:11:52.7638958Z GITHUB_ACTION=__self 2023-01-11T21:11:52.7639200Z VALGRIND=ON 2023-01-11T21:11:52.7639434Z GITHUB_REF_PROTECTED=false 2023-01-11T21:11:52.7639888Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2023-01-11T21:11:52.7640267Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2023-01-11T21:11:52.7640620Z *** 2023-01-11T21:11:52.7640855Z INSTALLED_VISION=yes 2023-01-11T21:11:52.7641099Z NVARCH=x86_64 2023-01-11T21:11:52.7641383Z NV_LIBCUSPARSE_DEV_VERSION=11.7.2.124-1 2023-01-11T21:11:52.7641667Z HOME=/var/lib/jenkins 2023-01-11T21:11:52.7642194Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_f3b45a2b-8801-4b5f-9961-1de1a8e6fa84 2023-01-11T21:11:52.7642598Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2023-01-11T21:11:52.7642857Z GITHUB_ACTION_REPOSITORY= 2023-01-11T21:11:52.7643122Z GITHUB_REF_TYPE=branch 2023-01-11T21:11:52.7643430Z NV_LIBNCCL_PACKAGE_VERSION=2.12.10-1 2023-01-11T21:11:52.7643693Z GITHUB_RETENTION_DAYS=90 2023-01-11T21:11:52.7644078Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2023-01-11T21:11:52.7644491Z NV_LIBNCCL_PACKAGE=libnccl2=2.12.10-1+cuda11.6 2023-01-11T21:11:52.7645022Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_f3b45a2b-8801-4b5f-9961-1de1a8e6fa84 2023-01-11T21:11:52.7645420Z DEBIAN_FRONTEND=noninteractive 2023-01-11T21:11:52.7645766Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2023-01-11T21:11:52.7646045Z GITHUB_REF=refs/pull/91627/merge 2023-01-11T21:11:52.7646350Z NV_CUDA_LIB_VERSION=11.6.2-1 2023-01-11T21:11:52.7646656Z GITHUB_SHA=57fc38f02f250896a12b32cfa200a6105a03d09c 2023-01-11T21:11:52.7646933Z INSTALLED_PROTOBUF=yes 2023-01-11T21:11:52.7647205Z GITHUB_REPOSITORY_ID=65600975 2023-01-11T21:11:52.7647469Z GITHUB_RUN_ID=3896099317 2023-01-11T21:11:52.7647796Z NV_LIBNPP_PACKAGE=libnpp-11-6=11.6.3.124-1 2023-01-11T21:11:52.7648099Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2023-01-11T21:11:52.7648396Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2023-01-11T21:11:52.7648709Z NV_NVTX_VERSION=11.6.124-1 2023-01-11T21:11:52.7648961Z CONTINUE_THROUGH_ERROR=False 2023-01-11T21:11:52.7649261Z GITHUB_SERVER_URL=https://github.com 2023-01-11T21:11:52.7649534Z MAX_JOBS=30 2023-01-11T21:11:52.7649756Z GITHUB_ACTOR_ID=7543386 2023-01-11T21:11:52.7650052Z NV_LIBCUBLAS_VERSION=11.9.2.110-1 2023-01-11T21:11:52.7650987Z NV_LIBCUBLAS_PACKAGE=libcublas-11-6=11.9.2.110-1 2023-01-11T21:11:52.7651470Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2023-01-11T21:11:52.7651930Z UCX_HOME=/usr 2023-01-11T21:11:52.7652183Z PYTORCH_RETRY_TEST_CASES=1 2023-01-11T21:11:52.7652496Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2023-01-11T21:11:52.7652843Z BASE_SHA=db2a237763eb8693a20788be94f8c192e762baa8 2023-01-11T21:11:52.7653185Z NV_CUDA_CUDART_DEV_VERSION=11.6.55-1 2023-01-11T21:11:52.7653472Z PR_BODY=Fixes #91003 cc @ezyang @gchanan 2023-01-11T21:11:52.7653746Z GITHUB_BASE_REF=master 2023-01-11T21:11:52.7653991Z TERM=xterm 2023-01-11T21:11:52.7654230Z TORCH_INDUCTOR_INSTALL_GXX=ON 2023-01-11T21:11:52.7654484Z XLA_CUDA= 2023-01-11T21:11:52.7654751Z NV_NVML_DEV_VERSION=11.6.55-1 2023-01-11T21:11:52.7655004Z TORCH_CUDA_ARCH_LIST=Maxwell 2023-01-11T21:11:52.7655264Z CUDA_VERSION=11.6.2 2023-01-11T21:11:52.7655602Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-6 2023-01-11T21:11:52.7655881Z OPENSSL_ROOT_DIR=/opt/openssl 2023-01-11T21:11:52.7656419Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_f3b45a2b-8801-4b5f-9961-1de1a8e6fa84 2023-01-11T21:11:52.7656804Z GITHUB_JOB=test 2023-01-11T21:11:52.7657060Z SCCACHE_S3_KEY_PREFIX=pull 2023-01-11T21:11:52.7657643Z COMMIT_MESSAGES=+ 52a16ce42647731c772e14e7175afa40fda07b3d make torchgen rename also Number arguments into input+ 87db01a53ecb702267ec36787654e418a52f8e93 fix torch.where signature mismatch+ 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e other instead of output in documentation 2023-01-11T21:11:52.7658242Z NVIDIA_DRIVER_CAPABILITIES=compute,utility 2023-01-11T21:11:52.7658527Z NUM_TEST_SHARDS=3 2023-01-11T21:11:52.7658832Z PR_NUMBER=91627 2023-01-11T21:11:52.7659379Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_f3b45a2b-8801-4b5f-9961-1de1a8e6fa84 2023-01-11T21:11:52.7659754Z SHLVL=1 2023-01-11T21:11:52.7660077Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-6 2023-01-11T21:11:52.7660403Z GITHUB_REPOSITORY=pytorch/pytorch 2023-01-11T21:11:52.7661190Z NVIDIA_REQUIRE_CUDA=cuda>=11.6 brand=tesla,driver>=418,driver<419 brand=tesla,driver>=450,driver<451 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 2023-01-11T21:11:52.7661950Z NV_LIBNPP_DEV_VERSION=11.6.3.124-1 2023-01-11T21:11:52.7662267Z SHA1=8419ddda87c8a47eacc63b54bc7ec98c1f27c26e 2023-01-11T21:11:52.7662558Z GITHUB_EVENT_NAME=pull_request 2023-01-11T21:11:52.7662866Z NV_CUDA_CUDART_VERSION=11.6.55-1 2023-01-11T21:11:52.7663210Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2023-01-11T21:11:52.7663481Z GITHUB_RUN_NUMBER=77928 2023-01-11T21:11:52.7663776Z GITHUB_WORKFLOW=pull 2023-01-11T21:11:52.7664189Z PATH=/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2023-01-11T21:11:52.7664623Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.12.10-1 2023-01-11T21:11:52.7664964Z GITHUB_WORKFLOW_SHA=57fc38f02f250896a12b32cfa200a6105a03d09c 2023-01-11T21:11:52.7665417Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2023-01-11T21:11:52.7665744Z GITHUB_TRIGGERING_ACTOR=albanD 2023-01-11T21:11:52.7666001Z _=/usr/bin/env 2023-01-11T21:11:52.7666283Z + echo 'Testing pytorch' 2023-01-11T21:11:52.7666521Z Testing pytorch 2023-01-11T21:11:52.7666791Z + export LANG=C.UTF-8 2023-01-11T21:11:52.7667053Z + LANG=C.UTF-8 2023-01-11T21:11:52.7667274Z + PR_NUMBER=91627 2023-01-11T21:11:52.7667539Z + [[ distributed == \d\e\f\a\u\l\t ]] 2023-01-11T21:11:52.7667837Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2023-01-11T21:11:52.7668243Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *rocm* ]] 2023-01-11T21:11:52.7668544Z + [[ distributed == \s\l\o\w ]] 2023-01-11T21:11:52.7668954Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *slow-gradcheck* ]] 2023-01-11T21:11:52.7669386Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *cuda* ]] 2023-01-11T21:11:52.7669781Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2023-01-11T21:11:52.7670096Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2023-01-11T21:11:52.7670388Z + [[ distributed == *crossref* ]] 2023-01-11T21:11:52.7670645Z + [[ distributed == *dynamo* ]] 2023-01-11T21:11:52.7670920Z + [[ distributed == *inductor* ]] 2023-01-11T21:11:52.7671303Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *rocm* ]] 2023-01-11T21:11:52.7671712Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 != *-bazel-* ]] 2023-01-11T21:11:52.7672091Z + pip_install --user ninja==1.10.2 2023-01-11T21:11:52.7672482Z + pip install --progress-bar off --user ninja==1.10.2 2023-01-11T21:11:53.3047356Z Collecting ninja==1.10.2 2023-01-11T21:11:53.3258344Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2023-01-11T21:11:54.1884229Z Installing collected packages: ninja 2023-01-11T21:11:54.1982468Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2023-01-11T21:11:54.1983143Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2023-01-11T21:11:54.2036206Z Successfully installed ninja-1.10.2 2023-01-11T21:11:54.2666965Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2023-01-11T21:11:54.2668517Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2023-01-11T21:11:54.2669687Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *asan* ]] 2023-01-11T21:11:54.2670523Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *-tsan* ]] 2023-01-11T21:11:54.2671216Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2023-01-11T21:11:54.2671835Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2023-01-11T21:11:54.2681530Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *tbb* ]] 2023-01-11T21:11:54.2698327Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *libtorch* ]] 2023-01-11T21:11:54.2698801Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *-bazel-* ]] 2023-01-11T21:11:54.2699238Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *-tsan* ]] 2023-01-11T21:11:54.2701427Z + cd test 2023-01-11T21:11:54.2702145Z + python -c 'import torch; print(torch.__config__.show())' 2023-01-11T21:11:55.9442425Z PyTorch built with: 2023-01-11T21:11:55.9442876Z - GCC 7.5 2023-01-11T21:11:55.9443170Z - C++ Version: 201703 2023-01-11T21:11:55.9443698Z - Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications 2023-01-11T21:11:55.9444263Z - Intel(R) MKL-DNN v2.7.2 (Git Hash fbec3e25a559ee252022ae066817b204e106a6ba) 2023-01-11T21:11:55.9444667Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2023-01-11T21:11:55.9445019Z - LAPACK is enabled (usually provided by MKL) 2023-01-11T21:11:55.9445344Z - NNPACK is enabled 2023-01-11T21:11:55.9445653Z - CPU capability usage: AVX2 2023-01-11T21:11:55.9445967Z - CUDA Runtime 11.6 2023-01-11T21:11:55.9446351Z - NVCC architecture flags: -gencode;arch=compute_52,code=sm_52 2023-01-11T21:11:55.9446744Z - CuDNN 8.3.2 (built against CUDA 11.5) 2023-01-11T21:11:55.9447041Z - Magma 2.6.1 2023-01-11T21:11:55.9450138Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.6, CUDNN_VERSION=8.3.2, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=2.0.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 2023-01-11T21:11:55.9453101Z 2023-01-11T21:11:56.1831145Z + cd test 2023-01-11T21:11:56.1831687Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2023-01-11T21:11:57.7734473Z ATen/Parallel: 2023-01-11T21:11:57.7734872Z at::get_num_threads() : 16 2023-01-11T21:11:57.7735323Z at::get_num_interop_threads() : 16 2023-01-11T21:11:57.7735978Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2023-01-11T21:11:57.7736503Z omp_get_max_threads() : 16 2023-01-11T21:11:57.7737194Z Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications 2023-01-11T21:11:57.7737585Z mkl_get_max_threads() : 16 2023-01-11T21:11:57.7738008Z Intel(R) MKL-DNN v2.7.2 (Git Hash fbec3e25a559ee252022ae066817b204e106a6ba) 2023-01-11T21:11:57.7738392Z std::thread::hardware_concurrency() : 32 2023-01-11T21:11:57.7738683Z Environment variables: 2023-01-11T21:11:57.7738957Z OMP_NUM_THREADS : [not set] 2023-01-11T21:11:57.7739480Z MKL_NUM_THREADS : [not set] 2023-01-11T21:11:57.7739787Z ATen parallel backend: OpenMP 2023-01-11T21:11:57.7739965Z 2023-01-11T21:11:58.0018889Z + [[ distributed == *backward* ]] 2023-01-11T21:11:58.0019526Z + [[ distributed == *xla* ]] 2023-01-11T21:11:58.0020177Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2023-01-11T21:11:58.0020827Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *libtorch* ]] 2023-01-11T21:11:58.0021150Z + [[ distributed == distributed ]] 2023-01-11T21:11:58.0021419Z + install_filelock 2023-01-11T21:11:58.0021695Z + pip_install filelock 2023-01-11T21:11:58.0022029Z + pip install --progress-bar off filelock 2023-01-11T21:11:58.5010657Z Collecting filelock 2023-01-11T21:11:58.5206483Z Downloading filelock-3.9.0-py3-none-any.whl (9.7 kB) 2023-01-11T21:11:59.3866784Z Installing collected packages: filelock 2023-01-11T21:11:59.4221876Z Successfully installed filelock-3.9.0 2023-01-11T21:11:59.4899827Z + install_triton 2023-01-11T21:11:59.4900110Z + local commit 2023-01-11T21:11:59.4900395Z + [[ distributed == *rocm* ]] 2023-01-11T21:11:59.4903995Z ++ get_pinned_commit triton 2023-01-11T21:11:59.4904286Z ++ cat .github/ci_commit_pins/triton.txt 2023-01-11T21:11:59.4918683Z + commit=0d7e7532279e45672555e344646f5c19c3972331 2023-01-11T21:11:59.4919534Z + pip_install --user git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python 2023-01-11T21:11:59.4920267Z + pip install --progress-bar off --user git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python 2023-01-11T21:11:59.9496325Z Collecting git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python 2023-01-11T21:11:59.9501947Z Cloning https://github.com/openai/triton (to revision 0d7e7532279e45672555e344646f5c19c3972331) to /tmp/pip-req-build-z9daxqwz 2023-01-11T21:11:59.9521759Z Running command git clone --filter=blob:none --quiet https://github.com/openai/triton /tmp/pip-req-build-z9daxqwz 2023-01-11T21:12:00.7190085Z Running command git rev-parse -q --verify 'sha^0d7e7532279e45672555e344646f5c19c3972331' 2023-01-11T21:12:00.7210632Z Running command git fetch -q https://github.com/openai/triton 0d7e7532279e45672555e344646f5c19c3972331 2023-01-11T21:12:01.1353998Z Running command git checkout -q 0d7e7532279e45672555e344646f5c19c3972331 2023-01-11T21:12:01.6406778Z Resolved https://github.com/openai/triton to commit 0d7e7532279e45672555e344646f5c19c3972331 2023-01-11T21:12:01.6407436Z Running command git submodule update --init --recursive -q 2023-01-11T21:12:02.2914965Z Preparing metadata (setup.py) ... [?25l- done 2023-01-11T21:12:02.5494382Z [?25hCollecting cmake 2023-01-11T21:12:02.5711696Z Downloading cmake-3.25.0-py2.py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (23.7 MB) 2023-01-11T21:12:02.9432463Z Requirement already satisfied: filelock in /opt/conda/lib/python3.10/site-packages (from triton==2.0.0) (3.9.0) 2023-01-11T21:12:02.9436078Z Requirement already satisfied: torch in /opt/conda/lib/python3.10/site-packages (from triton==2.0.0) (2.0.0a0+git8419ddd) 2023-01-11T21:12:02.9688056Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.10/site-packages (from torch->triton==2.0.0) (4.4.0) 2023-01-11T21:12:02.9692642Z Requirement already satisfied: networkx in /opt/conda/lib/python3.10/site-packages (from torch->triton==2.0.0) (2.6.3) 2023-01-11T21:12:02.9697784Z Requirement already satisfied: sympy in /opt/conda/lib/python3.10/site-packages (from torch->triton==2.0.0) (1.11.1) 2023-01-11T21:12:02.9908000Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.10/site-packages (from sympy->torch->triton==2.0.0) (1.2.1) 2023-01-11T21:12:02.9982010Z Building wheels for collected packages: triton 2023-01-11T21:12:55.4795856Z Building wheel for triton (setup.py) ... [?25l- \ | / - \ | / - \ | done 2023-01-11T21:12:55.5279141Z [?25h Created wheel for triton: filename=triton-2.0.0-cp310-cp310-linux_x86_64.whl size=15377935 sha256=6a98fea0ce89f0fdd0e10478f55779607056889cdd22f8989b683cd723c14179 2023-01-11T21:12:55.5280658Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/3f/1d/23/1c2bc47d618a44f9c949aea4b7e355e737a1f1ed208f009295 2023-01-11T21:12:55.5301615Z Successfully built triton 2023-01-11T21:12:56.4184670Z Installing collected packages: cmake, triton 2023-01-11T21:12:58.5770935Z Successfully installed cmake-3.25.0 triton-2.0.0 2023-01-11T21:12:58.6806654Z + pip_install --user jinja2 2023-01-11T21:12:58.6807064Z + pip install --progress-bar off --user jinja2 2023-01-11T21:12:59.7304900Z Collecting jinja2 2023-01-11T21:12:59.7578460Z Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB) 2023-01-11T21:13:00.0512081Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.10/site-packages (from jinja2) (2.1.1) 2023-01-11T21:13:00.9536591Z Installing collected packages: jinja2 2023-01-11T21:13:01.0581818Z Successfully installed jinja2-3.1.2 2023-01-11T21:13:01.1291890Z + test_distributed 2023-01-11T21:13:01.1292358Z + echo 'Testing distributed python tests' 2023-01-11T21:13:01.1295871Z Testing distributed python tests 2023-01-11T21:13:01.1296908Z + python test/run_test.py --distributed-tests --shard 1 3 --verbose 2023-01-11T21:13:03.3725257Z Ignoring disabled issues: ['91003'] 2023-01-11T21:13:03.4115257Z /var/lib/jenkins/workspace/test/run_test.py:1169: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. 2023-01-11T21:13:03.4115961Z if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6": 2023-01-11T21:13:03.4120656Z Found test time stats from artifacts 2023-01-11T21:13:03.4139129Z Selected tests: 2023-01-11T21:13:03.4139586Z distributed/algorithms/quantization/test_quantization 2023-01-11T21:13:03.4139924Z distributed/test_distributed_spawn 2023-01-11T21:13:03.4140215Z distributed/rpc/test_faulty_agent 2023-01-11T21:13:03.4140775Z distributed/pipeline/sync/test_stream 2023-01-11T21:13:03.4141094Z distributed/pipeline/sync/test_phony 2023-01-11T21:13:03.4141406Z distributed/pipeline/sync/test_dependency 2023-01-11T21:13:03.4141909Z distributed/pipeline/sync/test_checkpoint 2023-01-11T21:13:03.4142359Z distributed/pipeline/sync/skip/test_verify_skippables 2023-01-11T21:13:03.4145020Z distributed/pipeline/sync/skip/test_portal 2023-01-11T21:13:03.4145388Z distributed/pipeline/sync/skip/test_gpipe 2023-01-11T21:13:03.4145742Z distributed/optim/test_apply_optimizer_in_backward 2023-01-11T21:13:03.4146091Z distributed/elastic/events/lib_test 2023-01-11T21:13:03.4146570Z distributed/_shard/test_replicated_tensor 2023-01-11T21:13:03.4146894Z distributed/_composable/test_checkpoint 2023-01-11T21:13:03.4147221Z distributed/_tools/test_memory_tracker 2023-01-11T21:13:03.4147494Z distributed/nn/jit/test_instantiator 2023-01-11T21:13:03.4147804Z distributed/test_launcher 2023-01-11T21:13:03.4148082Z distributed/fsdp/test_utils 2023-01-11T21:13:03.4148338Z distributed/fsdp/test_fsdp_fx 2023-01-11T21:13:03.4148634Z distributed/_tensor/test_pointwise_ops 2023-01-11T21:13:03.4148959Z distributed/checkpoint/test_fsdp_optim_state 2023-01-11T21:13:03.4149274Z distributed/test_data_parallel 2023-01-11T21:13:03.4149558Z distributed/fsdp/test_fsdp_traversal 2023-01-11T21:13:03.4149857Z distributed/fsdp/test_fsdp_uneven 2023-01-11T21:13:03.4150188Z distributed/_shard/sharded_tensor/ops/test_embedding_bag 2023-01-11T21:13:03.4150530Z distributed/_shard/sharded_optim/test_sharded_optim 2023-01-11T21:13:03.4150879Z distributed/_shard/sharded_tensor/ops/test_init 2023-01-11T21:13:03.4151232Z distributed/_shard/sharding_spec/test_sharding_spec 2023-01-11T21:13:03.4151535Z distributed/fsdp/test_fsdp_input 2023-01-11T21:13:03.4151880Z distributed/tensor/parallel/test_2d_parallel 2023-01-11T21:13:03.4152203Z distributed/_tensor/test_math_ops 2023-01-11T21:13:03.4152497Z distributed/_shard/sharded_tensor/ops/test_tensor_ops 2023-01-11T21:13:03.4152820Z distributed/fsdp/test_fsdp_apply 2023-01-11T21:13:03.4153119Z distributed/_tensor/test_tp_sharding_ops 2023-01-11T21:13:03.4153551Z distributed/fsdp/test_fsdp_hybrid_shard 2023-01-11T21:13:03.4153858Z distributed/test_dynamo_distributed 2023-01-11T21:13:03.4154158Z distributed/_tensor/test_redistribute 2023-01-11T21:13:03.4154449Z distributed/fsdp/test_fsdp_ignored_modules 2023-01-11T21:13:03.4154735Z distributed/test_c10d_spawn_ucc 2023-01-11T21:13:03.4155021Z distributed/fsdp/test_fsdp_meta 2023-01-11T21:13:03.4155318Z distributed/fsdp/test_fsdp_exec_order 2023-01-11T21:13:03.4155623Z distributed/fsdp/test_fsdp_sharded_grad_scaler 2023-01-11T21:13:03.4155944Z distributed/_tensor/test_dtensor 2023-01-11T21:13:03.4156231Z distributed/algorithms/test_join 2023-01-11T21:13:03.4156518Z distributed/_composable/test_fully_shard 2023-01-11T21:13:03.4156816Z distributed/test_pg_wrapper 2023-01-11T21:13:03.4157111Z distributed/fsdp/test_fsdp_checkpoint 2023-01-11T21:13:03.4157383Z distributed/test_c10d_pypg 2023-01-11T21:13:03.4157684Z distributed/fsdp/test_fsdp_use_orig_params 2023-01-11T21:13:03.4158009Z distributed/fsdp/test_fsdp_mixed_precision 2023-01-11T21:13:03.4158330Z distributed/rpc/cuda/test_tensorpipe_agent 2023-01-11T21:13:03.4310012Z Prioritized test from test file changes. 2023-01-11T21:13:03.4310399Z reordering tests for PR: 2023-01-11T21:13:03.4310876Z prioritized: ['distributed/fsdp/test_fsdp_ignored_modules'] 2023-01-11T21:13:03.4315261Z the rest: ['distributed/algorithms/quantization/test_quantization', 'distributed/test_distributed_spawn', 'distributed/rpc/test_faulty_agent', 'distributed/pipeline/sync/test_stream', 'distributed/pipeline/sync/test_phony', 'distributed/pipeline/sync/test_dependency', 'distributed/pipeline/sync/test_checkpoint', 'distributed/pipeline/sync/skip/test_verify_skippables', 'distributed/pipeline/sync/skip/test_portal', 'distributed/pipeline/sync/skip/test_gpipe', 'distributed/optim/test_apply_optimizer_in_backward', 'distributed/elastic/events/lib_test', 'distributed/_shard/test_replicated_tensor', 'distributed/_composable/test_checkpoint', 'distributed/_tools/test_memory_tracker', 'distributed/nn/jit/test_instantiator', 'distributed/test_launcher', 'distributed/fsdp/test_utils', 'distributed/fsdp/test_fsdp_fx', 'distributed/_tensor/test_pointwise_ops', 'distributed/checkpoint/test_fsdp_optim_state', 'distributed/test_data_parallel', 'distributed/fsdp/test_fsdp_traversal', 'distributed/fsdp/test_fsdp_uneven', 'distributed/_shard/sharded_tensor/ops/test_embedding_bag', 'distributed/_shard/sharded_optim/test_sharded_optim', 'distributed/_shard/sharded_tensor/ops/test_init', 'distributed/_shard/sharding_spec/test_sharding_spec', 'distributed/fsdp/test_fsdp_input', 'distributed/tensor/parallel/test_2d_parallel', 'distributed/_tensor/test_math_ops', 'distributed/_shard/sharded_tensor/ops/test_tensor_ops', 'distributed/fsdp/test_fsdp_apply', 'distributed/_tensor/test_tp_sharding_ops', 'distributed/fsdp/test_fsdp_hybrid_shard', 'distributed/test_dynamo_distributed', 'distributed/_tensor/test_redistribute', 'distributed/test_c10d_spawn_ucc', 'distributed/fsdp/test_fsdp_meta', 'distributed/fsdp/test_fsdp_exec_order', 'distributed/fsdp/test_fsdp_sharded_grad_scaler', 'distributed/_tensor/test_dtensor', 'distributed/algorithms/test_join', 'distributed/_composable/test_fully_shard', 'distributed/test_pg_wrapper', 'distributed/fsdp/test_fsdp_checkpoint', 'distributed/test_c10d_pypg', 'distributed/fsdp/test_fsdp_use_orig_params', 'distributed/fsdp/test_fsdp_mixed_precision', 'distributed/rpc/cuda/test_tensorpipe_agent'] 2023-01-11T21:13:03.4318319Z 2023-01-11T21:13:03.4318862Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2023-01-11T21:13:03.4535017Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2023-01-11T21:13:03.4783318Z parallel (file granularity) tests: 2023-01-11T21:13:03.4783788Z 2023-01-11T21:13:03.4784122Z serial (file granularity) tests: 2023-01-11T21:13:03.4784579Z distributed/fsdp/test_fsdp_ignored_modules 2023-01-11T21:13:03.4784922Z distributed/algorithms/quantization/test_quantization 2023-01-11T21:13:03.4785266Z distributed/test_distributed_spawn 2023-01-11T21:13:03.4785563Z distributed/rpc/test_faulty_agent 2023-01-11T21:13:03.4785879Z distributed/pipeline/sync/test_stream 2023-01-11T21:13:03.4786162Z distributed/pipeline/sync/test_phony 2023-01-11T21:13:03.4786474Z distributed/pipeline/sync/test_dependency 2023-01-11T21:13:03.4786805Z distributed/pipeline/sync/test_checkpoint 2023-01-11T21:13:03.4787125Z distributed/pipeline/sync/skip/test_verify_skippables 2023-01-11T21:13:03.4787470Z distributed/pipeline/sync/skip/test_portal 2023-01-11T21:13:03.4787797Z distributed/pipeline/sync/skip/test_gpipe 2023-01-11T21:13:03.4788132Z distributed/optim/test_apply_optimizer_in_backward 2023-01-11T21:13:03.4788446Z distributed/elastic/events/lib_test 2023-01-11T21:13:03.4788751Z distributed/_shard/test_replicated_tensor 2023-01-11T21:13:03.4789072Z distributed/_composable/test_checkpoint 2023-01-11T21:13:03.4789362Z distributed/_tools/test_memory_tracker 2023-01-11T21:13:03.4789667Z distributed/nn/jit/test_instantiator 2023-01-11T21:13:03.4789952Z distributed/test_launcher 2023-01-11T21:13:03.4790207Z distributed/fsdp/test_utils 2023-01-11T21:13:03.4790486Z distributed/fsdp/test_fsdp_fx 2023-01-11T21:13:03.4790781Z distributed/_tensor/test_pointwise_ops 2023-01-11T21:13:03.4791089Z distributed/checkpoint/test_fsdp_optim_state 2023-01-11T21:13:03.4791399Z distributed/test_data_parallel 2023-01-11T21:13:03.4791691Z distributed/fsdp/test_fsdp_traversal 2023-01-11T21:13:03.4791972Z distributed/fsdp/test_fsdp_uneven 2023-01-11T21:13:03.4792306Z distributed/_shard/sharded_tensor/ops/test_embedding_bag 2023-01-11T21:13:03.4792667Z distributed/_shard/sharded_optim/test_sharded_optim 2023-01-11T21:13:03.4792995Z distributed/_shard/sharded_tensor/ops/test_init 2023-01-11T21:13:03.4793346Z distributed/_shard/sharding_spec/test_sharding_spec 2023-01-11T21:13:03.4793665Z distributed/fsdp/test_fsdp_input 2023-01-11T21:13:03.4793980Z distributed/tensor/parallel/test_2d_parallel 2023-01-11T21:13:03.4794271Z distributed/_tensor/test_math_ops 2023-01-11T21:13:03.4794595Z distributed/_shard/sharded_tensor/ops/test_tensor_ops 2023-01-11T21:13:03.4794912Z distributed/fsdp/test_fsdp_apply 2023-01-11T21:13:03.4795196Z distributed/_tensor/test_tp_sharding_ops 2023-01-11T21:13:03.4795512Z distributed/fsdp/test_fsdp_hybrid_shard 2023-01-11T21:13:03.4795908Z distributed/test_dynamo_distributed 2023-01-11T21:13:03.4796193Z distributed/_tensor/test_redistribute 2023-01-11T21:13:03.4796485Z distributed/test_c10d_spawn_ucc 2023-01-11T21:13:03.4796768Z distributed/fsdp/test_fsdp_meta 2023-01-11T21:13:03.4797045Z distributed/fsdp/test_fsdp_exec_order 2023-01-11T21:13:03.4797365Z distributed/fsdp/test_fsdp_sharded_grad_scaler 2023-01-11T21:13:03.4797673Z distributed/_tensor/test_dtensor 2023-01-11T21:13:03.4797990Z distributed/algorithms/test_join 2023-01-11T21:13:03.4798294Z distributed/_composable/test_fully_shard 2023-01-11T21:13:03.4798591Z distributed/test_pg_wrapper 2023-01-11T21:13:03.4798867Z distributed/fsdp/test_fsdp_checkpoint 2023-01-11T21:13:03.4799154Z distributed/test_c10d_pypg 2023-01-11T21:13:03.4799456Z distributed/fsdp/test_fsdp_use_orig_params 2023-01-11T21:13:03.4799763Z distributed/fsdp/test_fsdp_mixed_precision 2023-01-11T21:13:03.4800087Z distributed/rpc/cuda/test_tensorpipe_agent 2023-01-11T21:13:05.6404455Z Ignoring disabled issues: ['91003'] 2023-01-11T21:13:05.6532619Z Ignoring disabled issues: ['91003'] 2023-01-11T21:13:06.0507699Z Running distributed/fsdp/test_fsdp_ignored_modules ... [2023-01-11 21:13:06.050281] 2023-01-11T21:13:06.0512083Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_ignored_modules.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:13:06.050877] 2023-01-11T21:13:36.5916280Z 2023-01-11T21:13:36.5919071Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_ignored_modules 2023-01-11T21:13:36.5926226Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_ignored_modules (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_ignored_modules_tmrrv1ob) 2023-01-11T21:13:36.5926642Z 2023-01-11T21:13:36.5926761Z Running tests... 2023-01-11T21:13:36.5927290Z ---------------------------------------------------------------------- 2023-01-11T21:13:36.5927877Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_ignored_modules 2023-01-11T21:13:36.5928414Z test_diff_ignored_modules_across_ranks_pass_ignored_modules_to_root_False (__main__.TestFSDPIgnoredModules) 2023-01-11T21:13:36.5928912Z Tests ignoring different modules across ranks. ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:13:36.5929370Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 905 2023-01-11T21:13:36.5929923Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 906 2023-01-11T21:13:36.5930857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:13:36.5931316Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:13:36.5931895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:13:36.5932347Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:13:36.5932929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:13:36.5933367Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:13:36.5933938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:13:36.5934382Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:13:36.5934830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:13:36.5935323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:13:36.5935980Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:13:36.5936657Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:13:36.5937354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:13:36.5937814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:13:36.5938150Z dist init r=0, world=2 2023-01-11T21:13:36.5938401Z dist init r=1, world=2 2023-01-11T21:13:36.5938647Z ok (6.052s) 2023-01-11T21:13:36.5939063Z test_diff_ignored_modules_across_ranks_pass_ignored_modules_to_root_True (__main__.TestFSDPIgnoredModules) 2023-01-11T21:13:36.5939592Z Tests ignoring different modules across ranks. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 988 2023-01-11T21:13:36.5940543Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 989 2023-01-11T21:13:36.5941868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:13:36.5942554Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:13:36.5943125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:13:36.5943590Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:13:36.5944192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:13:36.5944612Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:13:36.5945303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:13:36.5945778Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:13:36.5946227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:13:36.5946695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:13:36.5947351Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:13:36.5948028Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:13:36.5948543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:13:36.5948994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:13:36.5949338Z dist init r=0, world=2 2023-01-11T21:13:36.5949587Z dist init r=1, world=2 2023-01-11T21:13:36.5949803Z ok (4.413s) 2023-01-11T21:13:36.5950121Z test_ignored_modules_invalid (__main__.TestFSDPIgnoredModules) 2023-01-11T21:13:36.5950634Z Tests that passing an FSDP module as an ignored module or the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1071 2023-01-11T21:13:36.5951133Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1072 2023-01-11T21:13:36.5951746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:13:36.5952191Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:13:36.5952759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:13:36.5953208Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:13:36.5953781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:13:36.5954220Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:13:36.5954783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:13:36.5955223Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:13:36.5955737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:13:36.5956225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:13:36.5956857Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:13:36.5957540Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:13:36.5958054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:13:36.5958521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:13:36.5958856Z dist init r=1, world=2 2023-01-11T21:13:36.5959106Z dist init r=0, world=2 2023-01-11T21:13:36.5959343Z ok (4.010s) 2023-01-11T21:13:36.5959645Z test_ignored_modules_nested (__main__.TestFSDPIgnoredModules) 2023-01-11T21:13:36.5960155Z Tests that passing a module with nested FSDP modules does not ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1150 2023-01-11T21:13:36.5960670Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1151 2023-01-11T21:13:36.5961271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:13:36.5961748Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:13:36.5962325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:13:36.5962786Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:13:36.5963403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:13:36.5963830Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:13:36.5964396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:13:36.5964854Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:13:36.5965298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:13:36.5965764Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:13:36.5966413Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:13:36.5967094Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:13:36.5967609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:13:36.5968055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:13:36.5968406Z dist init r=0, world=2 2023-01-11T21:13:36.5968655Z dist init r=1, world=2 2023-01-11T21:13:36.5968874Z ok (4.411s) 2023-01-11T21:13:36.5969334Z test_ignored_modules_not_under_wrapped_root (__main__.TestFSDPIgnoredModules) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1233 2023-01-11T21:13:36.5969884Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1234 2023-01-11T21:13:36.5971042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:13:36.5971483Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:13:36.5972057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:13:36.5972522Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:13:36.5973075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:13:36.5973617Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:13:36.5974183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:13:36.5974643Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:13:36.5975072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:13:36.5975557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:13:36.5976200Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:13:36.5976859Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:13:36.5977377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:13:36.5977840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:13:36.5978188Z dist init r=1, world=2 2023-01-11T21:13:36.5978421Z dist init r=0, world=2 2023-01-11T21:13:36.5978657Z ok (4.411s) 2023-01-11T21:13:36.5978991Z test_ignored_modules_transformer (__main__.TestFSDPIgnoredModules) 2023-01-11T21:13:36.5979690Z Tests that ignored modules' parameters are not flattened for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1316 2023-01-11T21:13:36.5980229Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1317 2023-01-11T21:13:36.5980830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:13:36.5981270Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:13:36.5981816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:13:36.5982283Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:13:36.5982853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:13:36.5983269Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:13:36.5983836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:13:36.5984291Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:13:36.5984740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:13:36.5985211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:13:36.5985860Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:13:36.5986548Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:13:36.5987062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:13:36.5987507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:13:36.5987854Z dist init r=0, world=2 2023-01-11T21:13:36.5988105Z dist init r=1, world=2 2023-01-11T21:13:36.5988326Z ok (4.912s) 2023-01-11T21:13:36.5988472Z 2023-01-11T21:13:36.5988749Z ---------------------------------------------------------------------- 2023-01-11T21:13:36.5989077Z Ran 6 tests in 28.210s 2023-01-11T21:13:36.5989237Z 2023-01-11T21:13:36.5989312Z OK 2023-01-11T21:13:36.5989443Z 2023-01-11T21:13:36.5989570Z Generating XML reports... 2023-01-11T21:13:36.5990197Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_ignored_modules/TEST-TestFSDPIgnoredModules-20230111211307.xml 2023-01-11T21:13:36.5990632Z 2023-01-11T21:13:36.5990952Z ##[endgroup] 2023-01-11T21:13:36.5991584Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_ignored_modules (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_ignored_modules_tmrrv1ob) 2023-01-11T21:13:36.5991956Z 2023-01-11T21:13:36.5992266Z Running distributed/algorithms/quantization/test_quantization ... [2023-01-11 21:13:36.591704] 2023-01-11T21:13:36.5992605Z /usr/bin/mpiexec 2023-01-11T21:13:36.5992951Z MPI not available -- MPI backend tests will be skipped 2023-01-11T21:13:36.5993523Z Map different backends to different shards for distributed/algorithms/quantization/test_quantization: {'gloo': 1, 'nccl': 2} 2023-01-11T21:13:36.5994006Z Running distributed tests for the test backend with env init_method in shard 1 of 3 2023-01-11T21:13:36.5994781Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:13:36.593810] 2023-01-11T21:13:38.6839692Z 2023-01-11T21:13:38.6840590Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization 2023-01-11T21:13:38.6842343Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_hgaetf6e) 2023-01-11T21:13:38.6843209Z 2023-01-11T21:13:38.6843425Z 2023-01-11T21:13:38.6843713Z ##[endgroup] 2023-01-11T21:13:38.6844520Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_hgaetf6e) 2023-01-11T21:13:38.6844953Z 2023-01-11T21:13:38.6851470Z Running distributed tests for the test backend with file init_method in shard 1 of 3 2023-01-11T21:13:38.6857074Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:13:38.685295] 2023-01-11T21:13:40.7720007Z 2023-01-11T21:13:40.7720785Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization 2023-01-11T21:13:40.7721970Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_p7um0863) 2023-01-11T21:13:40.7722575Z 2023-01-11T21:13:40.7722753Z 2023-01-11T21:13:40.7723054Z ##[endgroup] 2023-01-11T21:13:40.7723891Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_p7um0863) 2023-01-11T21:13:40.7724321Z 2023-01-11T21:13:40.7732345Z Shard 1: nccl should be run in 2 2023-01-11T21:13:40.7732713Z Running distributed tests for the gloo backend with env init_method in shard 1 of 3 2023-01-11T21:13:40.7736780Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:13:40.773326] 2023-01-11T21:14:03.6678943Z 2023-01-11T21:14:03.6679520Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization 2023-01-11T21:14:03.6681799Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_rlp35ude) 2023-01-11T21:14:03.6682966Z , <__main__.DistQuantizationTests testMethod=test_all_gather_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_fp16>]> 2023-01-11T21:14:03.6684054Z test_all_gather_bfp16 (__main__.DistQuantizationTests) 2023-01-11T21:14:03.6684404Z test_all_gather_fp16 (__main__.DistQuantizationTests) 2023-01-11T21:14:03.6684768Z test_all_to_all_bfp16 (__main__.DistQuantizationTests) 2023-01-11T21:14:03.6685128Z test_all_to_all_fp16 (__main__.DistQuantizationTests) 2023-01-11T21:14:03.6685481Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) 2023-01-11T21:14:03.6686169Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests) 2023-01-11T21:14:03.6686727Z 2023-01-11T21:14:03.6688074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:14:03.6688583Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:14:03.6689432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:14:03.6690165Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:14:03.6691024Z 2023-01-11T21:14:03.6691145Z Running tests... 2023-01-11T21:14:03.6691559Z ---------------------------------------------------------------------- 2023-01-11T21:14:03.6692314Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2023-01-11T21:14:03.6692872Z test_all_gather_bfp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:14:03.6693328Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1534 2023-01-11T21:14:03.6693784Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1535 2023-01-11T21:14:03.6694397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:14:03.6694846Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:14:03.6695399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:14:03.6695875Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:14:03.6696461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:14:03.6696903Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:14:03.6697454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:14:03.6697937Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:14:03.6698369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:14:03.6698847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:14:03.6699453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:14:03.6700268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:14:03.6701731Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:14:03.6702677Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:14:03.6703071Z ok (3.958s) 2023-01-11T21:14:03.6703227Z 2023-01-11T21:14:03.6703495Z ---------------------------------------------------------------------- 2023-01-11T21:14:03.6703826Z Ran 1 test in 3.958s 2023-01-11T21:14:03.6703968Z 2023-01-11T21:14:03.6704063Z OK 2023-01-11T21:14:03.6704198Z 2023-01-11T21:14:03.6704322Z Generating XML reports... 2023-01-11T21:14:03.6705587Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211344.xml 2023-01-11T21:14:03.6706886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:14:03.6707524Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:14:03.6708778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:14:03.6709548Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:14:03.6709798Z 2023-01-11T21:14:03.6709892Z Running tests... 2023-01-11T21:14:03.6710356Z ---------------------------------------------------------------------- 2023-01-11T21:14:03.6710963Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2023-01-11T21:14:03.6711496Z test_all_gather_fp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:14:03.6711973Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1643 2023-01-11T21:14:03.6712400Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1644 2023-01-11T21:14:03.6712988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:14:03.6713528Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:14:03.6714077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:14:03.6714554Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:14:03.6715395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:14:03.6715841Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:14:03.6716396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:14:03.6716856Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:14:03.6717287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:14:03.6717756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:14:03.6718220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:14:03.6718700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:14:03.6719344Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:14:03.6719995Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:14:03.6720386Z ok (3.958s) 2023-01-11T21:14:03.6720533Z 2023-01-11T21:14:03.6720800Z ---------------------------------------------------------------------- 2023-01-11T21:14:03.6721128Z Ran 1 test in 3.958s 2023-01-11T21:14:03.6721272Z 2023-01-11T21:14:03.6721364Z OK 2023-01-11T21:14:03.6721496Z 2023-01-11T21:14:03.6721618Z Generating XML reports... 2023-01-11T21:14:03.6722278Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211350.xml 2023-01-11T21:14:03.6723010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:14:03.6723449Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:14:03.6724014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:14:03.6724474Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:14:03.6724782Z 2023-01-11T21:14:03.6724874Z Running tests... 2023-01-11T21:14:03.6725273Z ---------------------------------------------------------------------- 2023-01-11T21:14:03.6725866Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2023-01-11T21:14:03.6726399Z test_all_to_all_bfp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_fp16 (0.001s) 2023-01-11T21:14:03.6726683Z 2023-01-11T21:14:03.6726944Z ---------------------------------------------------------------------- 2023-01-11T21:14:03.6727267Z Ran 1 test in 0.001s 2023-01-11T21:14:03.6727423Z 2023-01-11T21:14:03.6727529Z OK (skipped=1) 2023-01-11T21:14:03.6727679Z 2023-01-11T21:14:03.6727785Z Generating XML reports... 2023-01-11T21:14:03.6728433Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211356.xml 2023-01-11T21:14:03.6729178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:14:03.6729616Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:14:03.6730164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:14:03.6731294Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:14:03.6731522Z 2023-01-11T21:14:03.6731725Z Running tests... 2023-01-11T21:14:03.6732128Z ---------------------------------------------------------------------- 2023-01-11T21:14:03.6732724Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2023-01-11T21:14:03.6733272Z test_all_to_all_fp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_fp16 (0.001s) 2023-01-11T21:14:03.6733549Z 2023-01-11T21:14:03.6733813Z ---------------------------------------------------------------------- 2023-01-11T21:14:03.6734118Z Ran 1 test in 0.001s 2023-01-11T21:14:03.6734274Z 2023-01-11T21:14:03.6734381Z OK (skipped=1) 2023-01-11T21:14:03.6734532Z 2023-01-11T21:14:03.6734655Z Generating XML reports... 2023-01-11T21:14:03.6735288Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211359.xml 2023-01-11T21:14:03.6736034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:14:03.6736475Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:14:03.6737036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:14:03.6737477Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:14:03.6737699Z 2023-01-11T21:14:03.6737812Z Running tests... 2023-01-11T21:14:03.6738208Z ---------------------------------------------------------------------- 2023-01-11T21:14:03.6738801Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2023-01-11T21:14:03.6739357Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_single_bfp16 (0.001s) 2023-01-11T21:14:03.6739655Z 2023-01-11T21:14:03.6739918Z ---------------------------------------------------------------------- 2023-01-11T21:14:03.6740240Z Ran 1 test in 0.001s 2023-01-11T21:14:03.6740398Z 2023-01-11T21:14:03.6740488Z OK (skipped=1) 2023-01-11T21:14:03.6740638Z 2023-01-11T21:14:03.6740758Z Generating XML reports... 2023-01-11T21:14:03.6741402Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211401.xml 2023-01-11T21:14:03.6742145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:14:03.6742660Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:14:03.6743231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:14:03.6743690Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:14:03.6743912Z 2023-01-11T21:14:03.6744019Z Running tests... 2023-01-11T21:14:03.6744399Z ---------------------------------------------------------------------- 2023-01-11T21:14:03.6745019Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2023-01-11T21:14:03.6745585Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_single_fp16 (0.001s) 2023-01-11T21:14:03.6745878Z 2023-01-11T21:14:03.6746129Z ---------------------------------------------------------------------- 2023-01-11T21:14:03.6746445Z Ran 1 test in 0.001s 2023-01-11T21:14:03.6746604Z 2023-01-11T21:14:03.6746710Z OK (skipped=1) 2023-01-11T21:14:03.6746860Z 2023-01-11T21:14:03.6746983Z Generating XML reports... 2023-01-11T21:14:03.6747616Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211403.xml 2023-01-11T21:14:03.6748011Z 2023-01-11T21:14:03.6748350Z ##[endgroup] 2023-01-11T21:14:03.6749114Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_rlp35ude) 2023-01-11T21:14:03.6749553Z 2023-01-11T21:14:03.6749747Z Running distributed tests for the gloo backend with file init_method in shard 1 of 3 2023-01-11T21:14:03.6750521Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:14:03.669562] 2023-01-11T21:14:26.3042807Z 2023-01-11T21:14:26.3043377Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization 2023-01-11T21:14:26.3045955Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_z56oqcuj) 2023-01-11T21:14:26.3047145Z , <__main__.DistQuantizationTests testMethod=test_all_gather_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_fp16>]> 2023-01-11T21:14:26.3048001Z test_all_gather_bfp16 (__main__.DistQuantizationTests) 2023-01-11T21:14:26.3048366Z test_all_gather_fp16 (__main__.DistQuantizationTests) 2023-01-11T21:14:26.3048739Z test_all_to_all_bfp16 (__main__.DistQuantizationTests) 2023-01-11T21:14:26.3049097Z test_all_to_all_fp16 (__main__.DistQuantizationTests) 2023-01-11T21:14:26.3052138Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) 2023-01-11T21:14:26.3052530Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests) 2023-01-11T21:14:26.3052887Z 2023-01-11T21:14:26.3053552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:14:26.3053993Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:14:26.3054571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:14:26.3055056Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:14:26.3055275Z 2023-01-11T21:14:26.3055398Z Running tests... 2023-01-11T21:14:26.3055783Z ---------------------------------------------------------------------- 2023-01-11T21:14:26.3056607Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2023-01-11T21:14:26.3057161Z test_all_gather_bfp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:14:26.3057631Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1917 2023-01-11T21:14:26.3058085Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1918 2023-01-11T21:14:26.3058670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:14:26.3059113Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:14:26.3059675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:14:26.3060141Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:14:26.3060721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:14:26.3061167Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:14:26.3061696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:14:26.3062163Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:14:26.3062686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:14:26.3063152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:14:26.3063642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:14:26.3064117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:14:26.3064793Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:14:26.3065517Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:14:26.3065875Z ok (3.932s) 2023-01-11T21:14:26.3066026Z 2023-01-11T21:14:26.3066288Z ---------------------------------------------------------------------- 2023-01-11T21:14:26.3066606Z Ran 1 test in 3.932s 2023-01-11T21:14:26.3066767Z 2023-01-11T21:14:26.3066847Z OK 2023-01-11T21:14:26.3066979Z 2023-01-11T21:14:26.3067092Z Generating XML reports... 2023-01-11T21:14:26.3067746Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211407.xml 2023-01-11T21:14:26.3068492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:14:26.3068918Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:14:26.3069489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:14:26.3069957Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:14:26.3070175Z 2023-01-11T21:14:26.3070283Z Running tests... 2023-01-11T21:14:26.3070668Z ---------------------------------------------------------------------- 2023-01-11T21:14:26.3071278Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2023-01-11T21:14:26.3071800Z test_all_gather_fp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:14:26.3072242Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2026 2023-01-11T21:14:26.3072680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2027 2023-01-11T21:14:26.3073276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:14:26.3073791Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:14:26.3074346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:14:26.3074824Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:14:26.3075370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:14:26.3075807Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:14:26.3076353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:14:26.3076806Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:14:26.3077234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:14:26.3077681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:14:26.3078154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:14:26.3078635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:14:26.3079285Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:14:26.3080005Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:14:26.3080398Z ok (3.818s) 2023-01-11T21:14:26.3080544Z 2023-01-11T21:14:26.3080834Z ---------------------------------------------------------------------- 2023-01-11T21:14:26.3081122Z Ran 1 test in 3.818s 2023-01-11T21:14:26.3081279Z 2023-01-11T21:14:26.3081371Z OK 2023-01-11T21:14:26.3081501Z 2023-01-11T21:14:26.3081622Z Generating XML reports... 2023-01-11T21:14:26.3082279Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211413.xml 2023-01-11T21:14:26.3083011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:14:26.3083448Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:14:26.3084015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:14:26.3084477Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:14:26.3084685Z 2023-01-11T21:14:26.3084791Z Running tests... 2023-01-11T21:14:26.3085191Z ---------------------------------------------------------------------- 2023-01-11T21:14:26.3085788Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2023-01-11T21:14:26.3086322Z test_all_to_all_bfp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_fp16 (0.001s) 2023-01-11T21:14:26.3086604Z 2023-01-11T21:14:26.3086867Z ---------------------------------------------------------------------- 2023-01-11T21:14:26.3087185Z Ran 1 test in 0.001s 2023-01-11T21:14:26.3087342Z 2023-01-11T21:14:26.3087447Z OK (skipped=1) 2023-01-11T21:14:26.3087582Z 2023-01-11T21:14:26.3087703Z Generating XML reports... 2023-01-11T21:14:26.3088354Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211419.xml 2023-01-11T21:14:26.3089115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:14:26.3089523Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:14:26.3090088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:14:26.3091237Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:14:26.3091595Z 2023-01-11T21:14:26.3091685Z Running tests... 2023-01-11T21:14:26.3092082Z ---------------------------------------------------------------------- 2023-01-11T21:14:26.3092694Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2023-01-11T21:14:26.3093225Z test_all_to_all_fp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_fp16 (0.001s) 2023-01-11T21:14:26.3093505Z 2023-01-11T21:14:26.3093769Z ---------------------------------------------------------------------- 2023-01-11T21:14:26.3094071Z Ran 1 test in 0.001s 2023-01-11T21:14:26.3094228Z 2023-01-11T21:14:26.3094333Z OK (skipped=1) 2023-01-11T21:14:26.3094484Z 2023-01-11T21:14:26.3094605Z Generating XML reports... 2023-01-11T21:14:26.3095239Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211421.xml 2023-01-11T21:14:26.3095990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:14:26.3096426Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:14:26.3096988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:14:26.3097429Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:14:26.3097650Z 2023-01-11T21:14:26.3097833Z Running tests... 2023-01-11T21:14:26.3098239Z ---------------------------------------------------------------------- 2023-01-11T21:14:26.3098814Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2023-01-11T21:14:26.3099388Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_single_bfp16 (0.001s) 2023-01-11T21:14:26.3099689Z 2023-01-11T21:14:26.3099977Z ---------------------------------------------------------------------- 2023-01-11T21:14:26.3100282Z Ran 1 test in 0.001s 2023-01-11T21:14:26.3100439Z 2023-01-11T21:14:26.3100529Z OK (skipped=1) 2023-01-11T21:14:26.3100678Z 2023-01-11T21:14:26.3100799Z Generating XML reports... 2023-01-11T21:14:26.3101490Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211423.xml 2023-01-11T21:14:26.3102264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:14:26.3102688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:14:26.3103255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:14:26.3103712Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:14:26.3103937Z 2023-01-11T21:14:26.3104042Z Running tests... 2023-01-11T21:14:26.3104428Z ---------------------------------------------------------------------- 2023-01-11T21:14:26.3105038Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2023-01-11T21:14:26.3105591Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_single_fp16 (0.001s) 2023-01-11T21:14:26.3105883Z 2023-01-11T21:14:26.3106128Z ---------------------------------------------------------------------- 2023-01-11T21:14:26.3106453Z Ran 1 test in 0.001s 2023-01-11T21:14:26.3106610Z 2023-01-11T21:14:26.3106715Z OK (skipped=1) 2023-01-11T21:14:26.3106866Z 2023-01-11T21:14:26.3106987Z Generating XML reports... 2023-01-11T21:14:26.3107614Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211425.xml 2023-01-11T21:14:26.3108008Z 2023-01-11T21:14:26.3108432Z ##[endgroup] 2023-01-11T21:14:26.3109141Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_z56oqcuj) 2023-01-11T21:14:26.3109650Z 2023-01-11T21:14:26.3109843Z Running distributed tests for the ucc backend with env init_method in shard 1 of 3 2023-01-11T21:14:26.3110620Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:14:26.305691] 2023-01-11T21:14:28.4007464Z 2023-01-11T21:14:28.4008059Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization 2023-01-11T21:14:28.4009231Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_h4lir3k9) 2023-01-11T21:14:28.4009793Z 2023-01-11T21:14:28.4010007Z 2023-01-11T21:14:28.4010993Z ##[endgroup] 2023-01-11T21:14:28.4011826Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_h4lir3k9) 2023-01-11T21:14:28.4012254Z 2023-01-11T21:14:28.4019869Z Running distributed tests for the ucc backend with file init_method in shard 1 of 3 2023-01-11T21:14:28.4024492Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:14:28.402103] 2023-01-11T21:14:30.4396250Z 2023-01-11T21:14:30.4396750Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization 2023-01-11T21:14:30.4397824Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_hq0llu31) 2023-01-11T21:14:30.4398421Z 2023-01-11T21:14:30.4398618Z 2023-01-11T21:14:30.4398915Z ##[endgroup] 2023-01-11T21:14:30.4399712Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_hq0llu31) 2023-01-11T21:14:30.4400138Z 2023-01-11T21:14:30.4411068Z Running distributed/test_distributed_spawn ... [2023-01-11 21:14:30.440715] 2023-01-11T21:14:30.4419448Z /usr/bin/mpiexec 2023-01-11T21:14:30.4420802Z MPI not available -- MPI backend tests will be skipped 2023-01-11T21:14:30.4421940Z Map different backends to different shards for distributed/test_distributed_spawn: {'gloo': 1, 'nccl': 2, 'ucc': 3} 2023-01-11T21:14:30.4422899Z Running distributed tests for the test backend with env init_method in shard 1 of 3 2023-01-11T21:14:30.4428783Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:14:30.442575] 2023-01-11T21:14:32.8393689Z 2023-01-11T21:14:32.8394444Z Expand the folded group to see the log file of distributed/test_distributed_spawn 2023-01-11T21:14:32.8395355Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_7w0qpm95) 2023-01-11T21:14:32.8395855Z 2023-01-11T21:14:32.8396051Z 2023-01-11T21:14:32.8396366Z ##[endgroup] 2023-01-11T21:14:32.8397063Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_7w0qpm95) 2023-01-11T21:14:32.8397413Z 2023-01-11T21:14:32.8405266Z Running distributed tests for the test backend with file init_method in shard 1 of 3 2023-01-11T21:14:32.8410078Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:14:32.840717] 2023-01-11T21:14:35.1530069Z 2023-01-11T21:14:35.1531216Z Expand the folded group to see the log file of distributed/test_distributed_spawn 2023-01-11T21:14:35.1532121Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn__6g2y1pb) 2023-01-11T21:14:35.1532628Z 2023-01-11T21:14:35.1532821Z 2023-01-11T21:14:35.1534901Z ##[endgroup] 2023-01-11T21:14:35.1535607Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn__6g2y1pb) 2023-01-11T21:14:35.1535966Z 2023-01-11T21:14:35.1540847Z Shard 1: nccl should be run in 2 2023-01-11T21:14:35.1541377Z Running distributed tests for the gloo backend with env init_method in shard 1 of 3 2023-01-11T21:14:35.1546331Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:14:35.154343] 2023-01-11T21:44:22.5279454Z 2023-01-11T21:44:22.5280291Z Expand the folded group to see the log file of distributed/test_distributed_spawn 2023-01-11T21:44:22.5281270Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_amx6tdux) 2023-01-11T21:44:22.5289584Z 2023-01-11T21:44:22.5344234Z , <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_cat_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_stack_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_apply_optim_in_backward>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_apply_optim_in_backward_grad_as_bucket_view_false>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_apply_optim_in_backward_ignored_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_zero_output_features>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]> 2023-01-11T21:44:22.5387225Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5409576Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5410890Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5411722Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5412515Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5413286Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5413985Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5414626Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5415140Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5415675Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5416230Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5416771Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5417268Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5417795Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5418456Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5419391Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5420218Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5420644Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5421300Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5422113Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5422919Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5423851Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5424514Z test_all_gather (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5425296Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5426099Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5426899Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5427687Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5428238Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5428766Z test_all_gather_complex (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5429415Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5430032Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5430535Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5430934Z test_all_gather_group (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5431428Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5432173Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5432613Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5433127Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5433529Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5433954Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5434341Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5434745Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5435164Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5435641Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5436230Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5436801Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5437326Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5438035Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5438618Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5439375Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5440050Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5440473Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5440905Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5441517Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5442298Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5443118Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5443875Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5444656Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5445408Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5446154Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5446799Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5447517Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5448062Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5448806Z test_all_reduce_max (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5449524Z test_all_reduce_min (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5450128Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5451133Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5451748Z test_all_reduce_product (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5452466Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5453213Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5453947Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5454526Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5455260Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5456054Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5456847Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5457702Z test_all_to_all (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5458403Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5459097Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5459700Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5460092Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5460684Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5461165Z test_all_to_all_group (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5461530Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5461938Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5462370Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5462835Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5463679Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5464327Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5464779Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5465193Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5466031Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5466594Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5467422Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5468271Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5468709Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5469462Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5470324Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5470815Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5471666Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5472116Z test_average_parameters (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5472511Z test_backend_full_group (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5472888Z test_backend_group (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5473267Z test_barrier (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5473625Z test_barrier_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5474015Z test_barrier_full_group (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5474418Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5474771Z test_barrier_group (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5475145Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5475548Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5475940Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5476332Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5476741Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5477142Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5477517Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5477929Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5478332Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5478728Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5479212Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5479634Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5480055Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5480443Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5480826Z test_broadcast (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5481192Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5481559Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5481941Z test_broadcast_group (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5482322Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5482694Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5483143Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5483652Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5484109Z test_ddp_apply_optim_in_backward (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5484539Z test_ddp_apply_optim_in_backward_grad_as_bucket_view_false (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5485063Z test_ddp_apply_optim_in_backward_ignored_params (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5485501Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5485889Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5486298Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5486723Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5487165Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5487615Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5488054Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5488473Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5488888Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5489294Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5489661Z test_ddp_device (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5490028Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5491050Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5491476Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5491910Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5492339Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5492742Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5493152Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5493600Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5494096Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5494657Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5495256Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5495834Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5496535Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5497126Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5497726Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5498296Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5498890Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5499426Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5499916Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5500335Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5500717Z test_ddp_inference (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5501108Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5501508Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5501950Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5502376Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5502815Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5503251Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5503710Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5504126Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5504491Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5504900Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5505332Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5505758Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5506157Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5506568Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5506989Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5507395Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5507808Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5508214Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5508614Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5509011Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5509410Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5509825Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5510266Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5510695Z test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5511086Z test_destroy_full_group (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5511442Z test_destroy_group (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5511840Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5512255Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5512733Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5513094Z test_gather (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5513454Z test_gather_checks (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5513820Z test_gather_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5514178Z test_gather_full_group (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5514553Z test_gather_group (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5514918Z test_gather_object (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5515279Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5515658Z test_get_backend (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5516021Z test_get_future (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5516371Z test_get_rank (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5516731Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5517124Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5517508Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5517852Z test_irecv (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5518197Z test_isend (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5518568Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5518996Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5519415Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5519865Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5520309Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5520703Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5521115Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5521549Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5521952Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5522365Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5522776Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5523169Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5523573Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5523978Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5524367Z test_new_subgroups (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5524739Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5525199Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5525688Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5526131Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5526575Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5527025Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5527474Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5527881Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5528297Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5528724Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5529139Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5529641Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5530117Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5531287Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5531757Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5532177Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5532572Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5532952Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5533352Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5533736Z test_reduce_group_max (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5534112Z test_reduce_group_min (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5534484Z test_reduce_group_product (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5534866Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5535228Z test_reduce_max (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5535570Z test_reduce_min (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5535937Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5536311Z test_reduce_product (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5536773Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5537192Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5537561Z test_reduce_sum (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5537910Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5538289Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5538668Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5539035Z test_scatter (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5539379Z test_scatter_checks (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5539750Z test_scatter_complex (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5540117Z test_scatter_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5540476Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5540860Z test_scatter_full_group (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5541241Z test_scatter_group (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5541602Z test_scatter_object_list (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5541969Z test_send_recv (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5542339Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5542753Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5543175Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5543601Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5543990Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5544371Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5544792Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5545197Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5545571Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5545979Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5546407Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5546812Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5547191Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5547691Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5548085Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5548446Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5548858Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5549301Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5549741Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5550452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5550903Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5551474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5551924Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5552158Z 2023-01-11T21:44:22.5552270Z Running tests... 2023-01-11T21:44:22.5552676Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5553205Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.5553771Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.5554378Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2433 2023-01-11T21:44:22.5554825Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2434 2023-01-11T21:44:22.5555409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5555853Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5556422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5556888Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5557445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5557885Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5558447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5558903Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5559328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.5559816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.5560463Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5561133Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5561645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.5562112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.5562632Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2023-01-11T21:44:22.5563446Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2023-01-11T21:44:22.5564102Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2023-01-11T21:44:22.5564915Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2023-01-11T21:44:22.5565629Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2023-01-11T21:44:22.5566423Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2023-01-11T21:44:22.5567082Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2023-01-11T21:44:22.5567886Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2023-01-11T21:44:22.5568531Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2023-01-11T21:44:22.5569321Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2023-01-11T21:44:22.5569978Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2023-01-11T21:44:22.5571165Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2023-01-11T21:44:22.5571891Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2023-01-11T21:44:22.5572696Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2023-01-11T21:44:22.5573391Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2023-01-11T21:44:22.5574199Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2023-01-11T21:44:22.5574680Z ok (7.055s) 2023-01-11T21:44:22.5574831Z 2023-01-11T21:44:22.5575085Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5575415Z Ran 1 test in 7.055s 2023-01-11T21:44:22.5575574Z 2023-01-11T21:44:22.5575669Z OK 2023-01-11T21:44:22.5575800Z 2023-01-11T21:44:22.5575924Z Generating XML reports... 2023-01-11T21:44:22.5576516Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211439.xml 2023-01-11T21:44:22.5577224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5577670Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5578220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5578689Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5578916Z 2023-01-11T21:44:22.5579023Z Running tests... 2023-01-11T21:44:22.5579422Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5579933Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.5580460Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.003s) 2023-01-11T21:44:22.5580765Z 2023-01-11T21:44:22.5581032Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5581359Z Ran 1 test in 0.004s 2023-01-11T21:44:22.5581501Z 2023-01-11T21:44:22.5581609Z OK (skipped=1) 2023-01-11T21:44:22.5581761Z 2023-01-11T21:44:22.5581883Z Generating XML reports... 2023-01-11T21:44:22.5582480Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211448.xml 2023-01-11T21:44:22.5583254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5583698Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5584263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5584731Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5584937Z 2023-01-11T21:44:22.5585046Z Running tests... 2023-01-11T21:44:22.5585449Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5585975Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.5586458Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.5586938Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2578 2023-01-11T21:44:22.5587380Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2579 2023-01-11T21:44:22.5587982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5588409Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5589039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5589512Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5590066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5590502Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5591064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5591522Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5591951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.5592440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.5593095Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5593777Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5594270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.5594732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.5595073Z ok (4.108s) 2023-01-11T21:44:22.5595221Z 2023-01-11T21:44:22.5595476Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5595802Z Ran 1 test in 4.108s 2023-01-11T21:44:22.5595961Z 2023-01-11T21:44:22.5596053Z OK 2023-01-11T21:44:22.5596184Z 2023-01-11T21:44:22.5596308Z Generating XML reports... 2023-01-11T21:44:22.5596891Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211451.xml 2023-01-11T21:44:22.5597603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5598046Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5598592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5599056Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5599282Z 2023-01-11T21:44:22.5599390Z Running tests... 2023-01-11T21:44:22.5599853Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5600364Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.5600886Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.5601935Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.599s) 2023-01-11T21:44:22.5602457Z 2023-01-11T21:44:22.5602720Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5603027Z Ran 1 test in 1.599s 2023-01-11T21:44:22.5603188Z 2023-01-11T21:44:22.5603295Z OK (skipped=1) 2023-01-11T21:44:22.5603446Z 2023-01-11T21:44:22.5603569Z Generating XML reports... 2023-01-11T21:44:22.5604175Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211457.xml 2023-01-11T21:44:22.5604863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5605306Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5605871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5606390Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5606605Z 2023-01-11T21:44:22.5606713Z Running tests... 2023-01-11T21:44:22.5607112Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5607635Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.5608145Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.5608655Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2721 2023-01-11T21:44:22.5609094Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2722 2023-01-11T21:44:22.5609691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5610121Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5611054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5611518Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5612070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5612504Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5613067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5613527Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5613954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.5614442Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.5615094Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5615777Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5616273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.5616734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.5617288Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5617748Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5618093Z ok (4.191s) 2023-01-11T21:44:22.5618241Z 2023-01-11T21:44:22.5618517Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5618843Z Ran 1 test in 4.191s 2023-01-11T21:44:22.5618986Z 2023-01-11T21:44:22.5619079Z OK 2023-01-11T21:44:22.5619215Z 2023-01-11T21:44:22.5619338Z Generating XML reports... 2023-01-11T21:44:22.5619942Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211502.xml 2023-01-11T21:44:22.5620626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5621069Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5621634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5622098Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5622305Z 2023-01-11T21:44:22.5622414Z Running tests... 2023-01-11T21:44:22.5622813Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5623338Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.5623930Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.5624466Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2834 2023-01-11T21:44:22.5624906Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2835 2023-01-11T21:44:22.5625505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5625936Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5626502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5626961Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5627513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5627950Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5628510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5628963Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5629388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.5629871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.5630521Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5631195Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5631691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.5632157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.5632633Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5633090Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5633437Z ok (4.348s) 2023-01-11T21:44:22.5633581Z 2023-01-11T21:44:22.5633853Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5634237Z Ran 1 test in 4.349s 2023-01-11T21:44:22.5634378Z 2023-01-11T21:44:22.5634469Z OK 2023-01-11T21:44:22.5634601Z 2023-01-11T21:44:22.5634724Z Generating XML reports... 2023-01-11T21:44:22.5635324Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211508.xml 2023-01-11T21:44:22.5636017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5636466Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5637030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5637490Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5637697Z 2023-01-11T21:44:22.5637807Z Running tests... 2023-01-11T21:44:22.5638208Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5638738Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.5639258Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.5639780Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2947 2023-01-11T21:44:22.5640217Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2948 2023-01-11T21:44:22.5640869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5641302Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5641868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5642328Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5642903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5643327Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5643891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5644343Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5644776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.5645261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.5645906Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5646581Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5647077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.5647542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.5648015Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5648489Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5648948Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5649417Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5649883Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5650645Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5651123Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5651665Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5652129Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5652581Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5653042Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5653503Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5653828Z ok (6.728s) 2023-01-11T21:44:22.5653975Z 2023-01-11T21:44:22.5654255Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5654580Z Ran 1 test in 6.728s 2023-01-11T21:44:22.5654737Z 2023-01-11T21:44:22.5654828Z OK 2023-01-11T21:44:22.5654943Z 2023-01-11T21:44:22.5655064Z Generating XML reports... 2023-01-11T21:44:22.5655667Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211515.xml 2023-01-11T21:44:22.5656380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5656806Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5657372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5657897Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5658132Z 2023-01-11T21:44:22.5658240Z Running tests... 2023-01-11T21:44:22.5658624Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5659149Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.5659695Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.5660212Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3062 2023-01-11T21:44:22.5660649Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3063 2023-01-11T21:44:22.5661238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5661679Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5662232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5662691Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5663260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5663678Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5664242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5664766Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5665215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.5665688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.5666340Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5667020Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5667529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.5667976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.5668441Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5668999Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5669457Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5669928Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5670270Z ok (5.793s) 2023-01-11T21:44:22.5670418Z 2023-01-11T21:44:22.5670691Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5671001Z Ran 1 test in 5.793s 2023-01-11T21:44:22.5671158Z 2023-01-11T21:44:22.5671251Z OK 2023-01-11T21:44:22.5671383Z 2023-01-11T21:44:22.5671505Z Generating XML reports... 2023-01-11T21:44:22.5672091Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211524.xml 2023-01-11T21:44:22.5672795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5673290Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5673873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5674310Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5674536Z 2023-01-11T21:44:22.5674644Z Running tests... 2023-01-11T21:44:22.5675112Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5675624Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.5676186Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.5676720Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3177 2023-01-11T21:44:22.5677165Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3178 2023-01-11T21:44:22.5677759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5678196Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5678759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5679225Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5679780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5680220Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5680782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5681220Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5681671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.5682158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.5682813Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5683480Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5683996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.5684463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.5684932Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5685389Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5685927Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5686399Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5686850Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5687319Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5687786Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5688245Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5688570Z ok (5.823s) 2023-01-11T21:44:22.5688713Z 2023-01-11T21:44:22.5688989Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5689317Z Ran 1 test in 5.823s 2023-01-11T21:44:22.5689477Z 2023-01-11T21:44:22.5689551Z OK 2023-01-11T21:44:22.5689687Z 2023-01-11T21:44:22.5689809Z Generating XML reports... 2023-01-11T21:44:22.5690844Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211533.xml 2023-01-11T21:44:22.5691571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5691999Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5692643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5693120Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5693348Z 2023-01-11T21:44:22.5693440Z Running tests... 2023-01-11T21:44:22.5693846Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5694374Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.5694959Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.5695494Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3292 2023-01-11T21:44:22.5695931Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3293 2023-01-11T21:44:22.5696536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5696980Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5697531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5697992Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5698563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5698986Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5699553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5700010Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5700452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.5700925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.5701571Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5702254Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5702745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.5703294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.5703759Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5704235Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5704566Z ok (6.063s) 2023-01-11T21:44:22.5704711Z 2023-01-11T21:44:22.5704984Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5705318Z Ran 1 test in 6.063s 2023-01-11T21:44:22.5705477Z 2023-01-11T21:44:22.5705551Z OK 2023-01-11T21:44:22.5705685Z 2023-01-11T21:44:22.5705808Z Generating XML reports... 2023-01-11T21:44:22.5706412Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211541.xml 2023-01-11T21:44:22.5707117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5707599Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5708167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5708631Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5708858Z 2023-01-11T21:44:22.5708966Z Running tests... 2023-01-11T21:44:22.5709354Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5709934Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.5710518Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.5711049Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3407 2023-01-11T21:44:22.5711487Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3408 2023-01-11T21:44:22.5712094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5712537Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5713091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5713552Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5714128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5714551Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5715118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5715574Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5716018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.5716492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.5717140Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5717817Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5718336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.5718784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.5719248Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5719729Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5720250Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5720722Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5721063Z ok (6.614s) 2023-01-11T21:44:22.5721209Z 2023-01-11T21:44:22.5721484Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5721796Z Ran 1 test in 6.614s 2023-01-11T21:44:22.5721953Z 2023-01-11T21:44:22.5722046Z OK 2023-01-11T21:44:22.5722176Z 2023-01-11T21:44:22.5722302Z Generating XML reports... 2023-01-11T21:44:22.5722890Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211550.xml 2023-01-11T21:44:22.5723596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5724038Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5724605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5725052Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5725280Z 2023-01-11T21:44:22.5725387Z Running tests... 2023-01-11T21:44:22.5725789Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5726298Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.5726911Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.5727451Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3522 2023-01-11T21:44:22.5727889Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3523 2023-01-11T21:44:22.5728473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5728920Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5729486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5729950Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5730745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5731202Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5731772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5732223Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5732664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.5733154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.5733808Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5734469Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5734979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.5735448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.5735902Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5736377Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5736847Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5737317Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5737731Z ok (6.244s) 2023-01-11T21:44:22.5737876Z 2023-01-11T21:44:22.5738149Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5738476Z Ran 1 test in 6.244s 2023-01-11T21:44:22.5738635Z 2023-01-11T21:44:22.5738709Z OK 2023-01-11T21:44:22.5738841Z 2023-01-11T21:44:22.5738963Z Generating XML reports... 2023-01-11T21:44:22.5739570Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211559.xml 2023-01-11T21:44:22.5740278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5740705Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5741273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5741735Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5741965Z 2023-01-11T21:44:22.5742074Z Running tests... 2023-01-11T21:44:22.5742457Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5742982Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.5743559Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.5744156Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3637 2023-01-11T21:44:22.5744601Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3638 2023-01-11T21:44:22.5745199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5745643Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5746194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5746660Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5747229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5747648Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5748215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5748673Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5749116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.5749588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.5750236Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5750923Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5751433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.5751878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.5752344Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5752819Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5753273Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5753743Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5754084Z ok (5.707s) 2023-01-11T21:44:22.5754230Z 2023-01-11T21:44:22.5754563Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5754873Z Ran 1 test in 5.707s 2023-01-11T21:44:22.5755034Z 2023-01-11T21:44:22.5755127Z OK 2023-01-11T21:44:22.5755260Z 2023-01-11T21:44:22.5755383Z Generating XML reports... 2023-01-11T21:44:22.5755966Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211608.xml 2023-01-11T21:44:22.5756677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5757123Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5757691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5758137Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5758363Z 2023-01-11T21:44:22.5758470Z Running tests... 2023-01-11T21:44:22.5758875Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5759386Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.5759930Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.5761039Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.630s) 2023-01-11T21:44:22.5761563Z 2023-01-11T21:44:22.5761833Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5762160Z Ran 1 test in 1.631s 2023-01-11T21:44:22.5762302Z 2023-01-11T21:44:22.5762410Z OK (skipped=1) 2023-01-11T21:44:22.5762561Z 2023-01-11T21:44:22.5762685Z Generating XML reports... 2023-01-11T21:44:22.5763287Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211616.xml 2023-01-11T21:44:22.5763971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5764417Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5764990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5765453Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5765659Z 2023-01-11T21:44:22.5765767Z Running tests... 2023-01-11T21:44:22.5766169Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5766695Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.5767212Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.5767732Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3786 2023-01-11T21:44:22.5768171Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3787 2023-01-11T21:44:22.5768763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5769189Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5769762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5770223Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5771026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5771466Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5772124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5772582Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5773013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.5773536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.5774189Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5774869Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5775366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.5775829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.5776173Z ok (4.208s) 2023-01-11T21:44:22.5776319Z 2023-01-11T21:44:22.5776573Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5776900Z Ran 1 test in 4.208s 2023-01-11T21:44:22.5777058Z 2023-01-11T21:44:22.5777151Z OK 2023-01-11T21:44:22.5777283Z 2023-01-11T21:44:22.5777407Z Generating XML reports... 2023-01-11T21:44:22.5778055Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211620.xml 2023-01-11T21:44:22.5778780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5779225Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5779773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5780236Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5780467Z 2023-01-11T21:44:22.5780573Z Running tests... 2023-01-11T21:44:22.5780973Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5781482Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.5782034Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.5783096Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.638s) 2023-01-11T21:44:22.5783609Z 2023-01-11T21:44:22.5783875Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5784183Z Ran 1 test in 1.638s 2023-01-11T21:44:22.5784340Z 2023-01-11T21:44:22.5784453Z OK (skipped=1) 2023-01-11T21:44:22.5784607Z 2023-01-11T21:44:22.5784731Z Generating XML reports... 2023-01-11T21:44:22.5785317Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211627.xml 2023-01-11T21:44:22.5786020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5786463Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5787032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5787479Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5787705Z 2023-01-11T21:44:22.5787812Z Running tests... 2023-01-11T21:44:22.5788211Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5788738Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.5789317Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.5789814Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3929 2023-01-11T21:44:22.5790250Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3930 2023-01-11T21:44:22.5790839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5791280Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5791848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5792312Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5792864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5793307Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5793869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5794307Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5794751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.5795291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.5795952Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5796611Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5797124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.5797595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.5797935Z ok (5.716s) 2023-01-11T21:44:22.5798064Z 2023-01-11T21:44:22.5798332Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5798658Z Ran 1 test in 5.716s 2023-01-11T21:44:22.5798820Z 2023-01-11T21:44:22.5798913Z OK 2023-01-11T21:44:22.5799044Z 2023-01-11T21:44:22.5799150Z Generating XML reports... 2023-01-11T21:44:22.5799751Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211631.xml 2023-01-11T21:44:22.5800462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5800906Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5801459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5801920Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5802146Z 2023-01-11T21:44:22.5802253Z Running tests... 2023-01-11T21:44:22.5802636Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5803160Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.5803661Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s) 2023-01-11T21:44:22.5803944Z 2023-01-11T21:44:22.5804209Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5804518Z Ran 1 test in 0.002s 2023-01-11T21:44:22.5804676Z 2023-01-11T21:44:22.5804782Z OK (skipped=1) 2023-01-11T21:44:22.5804933Z 2023-01-11T21:44:22.5805055Z Generating XML reports... 2023-01-11T21:44:22.5805630Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211639.xml 2023-01-11T21:44:22.5806397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5806838Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5807406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5807847Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5808073Z 2023-01-11T21:44:22.5808184Z Running tests... 2023-01-11T21:44:22.5808586Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5809096Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.5809547Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5810018Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.5810722Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4073 2023-01-11T21:44:22.5811146Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4074 2023-01-11T21:44:22.5811744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5812187Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5812809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5813288Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5813863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5814302Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5814851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5815311Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5815756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.5816244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.5816880Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5817562Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5818077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.5818542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.5818868Z ok (4.226s) 2023-01-11T21:44:22.5819020Z 2023-01-11T21:44:22.5819291Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5819617Z Ran 1 test in 4.226s 2023-01-11T21:44:22.5819757Z 2023-01-11T21:44:22.5819850Z OK 2023-01-11T21:44:22.5819983Z 2023-01-11T21:44:22.5820105Z Generating XML reports... 2023-01-11T21:44:22.5820708Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211642.xml 2023-01-11T21:44:22.5821403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5821848Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5822417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5822879Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5823105Z 2023-01-11T21:44:22.5823273Z Running tests... 2023-01-11T21:44:22.5823679Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5824206Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.5824663Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5825155Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.5825634Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4186 2023-01-11T21:44:22.5826072Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4187 2023-01-11T21:44:22.5826653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5827095Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5827664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5828131Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5828684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5829120Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5829737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5830187Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5830631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.5831113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.5831768Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5832434Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5832948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.5833410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.5833882Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5834342Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5834688Z ok (4.239s) 2023-01-11T21:44:22.5834834Z 2023-01-11T21:44:22.5835106Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5835417Z Ran 1 test in 4.239s 2023-01-11T21:44:22.5835576Z 2023-01-11T21:44:22.5835667Z OK 2023-01-11T21:44:22.5835799Z 2023-01-11T21:44:22.5835921Z Generating XML reports... 2023-01-11T21:44:22.5836508Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211649.xml 2023-01-11T21:44:22.5837218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5837660Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5880291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5880858Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5881074Z 2023-01-11T21:44:22.5881184Z Running tests... 2023-01-11T21:44:22.5881633Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5882165Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.5882638Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5883327Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.5883822Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4299 2023-01-11T21:44:22.5884260Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4300 2023-01-11T21:44:22.5884859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5885305Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5885872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5886314Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5886884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5887323Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5887880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5888319Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5888766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.5889329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.5889995Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5890956Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5891476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.5891945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.5892399Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5892871Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.5893215Z ok (4.243s) 2023-01-11T21:44:22.5893361Z 2023-01-11T21:44:22.5893635Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5893944Z Ran 1 test in 4.243s 2023-01-11T21:44:22.5894103Z 2023-01-11T21:44:22.5894195Z OK 2023-01-11T21:44:22.5894326Z 2023-01-11T21:44:22.5894450Z Generating XML reports... 2023-01-11T21:44:22.5895033Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211655.xml 2023-01-11T21:44:22.5895741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5896188Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5896756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5897200Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5897424Z 2023-01-11T21:44:22.5897530Z Running tests... 2023-01-11T21:44:22.5897933Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5898438Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.5898905Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn) 2023-01-11T21:44:22.5899385Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.5899853Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4412 2023-01-11T21:44:22.5900372Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4413 2023-01-11T21:44:22.5900963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5901397Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5901945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5902411Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5902977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5903412Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5903951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5904403Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5904850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.5905342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.5905968Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5906714Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5907240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.5907684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.5908021Z ok (4.241s) 2023-01-11T21:44:22.5908169Z 2023-01-11T21:44:22.5908441Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5908770Z Ran 1 test in 4.242s 2023-01-11T21:44:22.5908912Z 2023-01-11T21:44:22.5909004Z OK 2023-01-11T21:44:22.5909135Z 2023-01-11T21:44:22.5909258Z Generating XML reports... 2023-01-11T21:44:22.5909857Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211702.xml 2023-01-11T21:44:22.5910539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5910985Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5911546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5912006Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5912231Z 2023-01-11T21:44:22.5912321Z Running tests... 2023-01-11T21:44:22.5912717Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5913246Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.5913714Z test_all_gather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.5914177Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4525 2023-01-11T21:44:22.5914620Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4526 2023-01-11T21:44:22.5915218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5915640Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5916204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5916664Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5917232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5917728Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5918291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5918741Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5919170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.5919659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.5920307Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5920984Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5921475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.5921941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.5922706Z STAGE:2023-01-11 21:17:13 4525:4525 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:17:13 4526:4526 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.5923075Z 2023-01-11T21:44:22.5923658Z STAGE:2023-01-11 21:17:13 4526:4526 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:17:13 4525:4525 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.5924044Z 2023-01-11T21:44:22.5924582Z STAGE:2023-01-11 21:17:13 4525:4525 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:17:13 4526:4526 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.5924970Z 2023-01-11T21:44:22.5925298Z STAGE:2023-01-11 21:17:13 4526:4526 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.5925855Z STAGE:2023-01-11 21:17:13 4525:4525 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.5926417Z STAGE:2023-01-11 21:17:13 4525:4525 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.5926958Z STAGE:2023-01-11 21:17:13 4526:4526 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.5927541Z STAGE:2023-01-11 21:17:13 4525:4525 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.5928127Z STAGE:2023-01-11 21:17:13 4526:4526 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.5928475Z ok (4.252s) 2023-01-11T21:44:22.5928621Z 2023-01-11T21:44:22.5928874Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5929198Z Ran 1 test in 4.252s 2023-01-11T21:44:22.5929356Z 2023-01-11T21:44:22.5929448Z OK 2023-01-11T21:44:22.5929585Z 2023-01-11T21:44:22.5929691Z Generating XML reports... 2023-01-11T21:44:22.5930531Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211709.xml 2023-01-11T21:44:22.5931256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5931697Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5932246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5932705Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5932932Z 2023-01-11T21:44:22.5933039Z Running tests... 2023-01-11T21:44:22.5933442Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5933952Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.5934557Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.5935048Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4638 2023-01-11T21:44:22.5935469Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4639 2023-01-11T21:44:22.5936065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5936511Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5937076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5937519Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5938085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5938519Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5939065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5939523Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5939966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.5940518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.5941161Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5941838Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5942346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.5942808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.5943359Z STAGE:2023-01-11 21:17:20 4639:4639 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.5943922Z STAGE:2023-01-11 21:17:20 4638:4638 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.5944900Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2588: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T21:44:22.5945520Z warnings.warn( 2023-01-11T21:44:22.5946356Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2588: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T21:44:22.5946966Z warnings.warn( 2023-01-11T21:44:22.5947429Z STAGE:2023-01-11 21:17:20 4639:4639 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.5947998Z STAGE:2023-01-11 21:17:20 4638:4638 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.5948560Z STAGE:2023-01-11 21:17:20 4639:4639 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.5949153Z STAGE:2023-01-11 21:17:20 4638:4638 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.5949722Z STAGE:2023-01-11 21:17:20 4639:4639 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.5950263Z STAGE:2023-01-11 21:17:20 4638:4638 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.5950824Z STAGE:2023-01-11 21:17:20 4638:4638 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.5951388Z STAGE:2023-01-11 21:17:20 4639:4639 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.5952042Z STAGE:2023-01-11 21:17:20 4638:4638 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.5952604Z STAGE:2023-01-11 21:17:20 4639:4639 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.5953160Z STAGE:2023-01-11 21:17:20 4638:4638 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.5953716Z STAGE:2023-01-11 21:17:20 4639:4639 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.5954275Z STAGE:2023-01-11 21:17:20 4639:4639 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.5955039Z STAGE:2023-01-11 21:17:20 4639:4639 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:17:20 4638:4638 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.5955421Z 2023-01-11T21:44:22.5955756Z STAGE:2023-01-11 21:17:20 4638:4638 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.5956110Z ok (4.265s) 2023-01-11T21:44:22.5956255Z 2023-01-11T21:44:22.5956525Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5956832Z Ran 1 test in 4.265s 2023-01-11T21:44:22.5956990Z 2023-01-11T21:44:22.5957082Z OK 2023-01-11T21:44:22.5957215Z 2023-01-11T21:44:22.5957338Z Generating XML reports... 2023-01-11T21:44:22.5957974Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211716.xml 2023-01-11T21:44:22.5958686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5959129Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5959696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5960139Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5960163Z 2023-01-11T21:44:22.5960270Z Running tests... 2023-01-11T21:44:22.5960534Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5960844Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.5961119Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.5961338Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4751 2023-01-11T21:44:22.5961550Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4752 2023-01-11T21:44:22.5961900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5962072Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5962446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5962636Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5962994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5963163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5963529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5963714Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5963940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.5964178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.5964573Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5965022Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5965246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.5965470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.5965708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T21:44:22.5965943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T21:44:22.5966335Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.5966722Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.5967036Z STAGE:2023-01-11 21:17:26 4752:4752 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.5967358Z STAGE:2023-01-11 21:17:26 4751:4751 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.5968141Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2588: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T21:44:22.5968260Z warnings.warn( 2023-01-11T21:44:22.5968981Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2588: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T21:44:22.5969090Z warnings.warn( 2023-01-11T21:44:22.5969417Z STAGE:2023-01-11 21:17:26 4752:4752 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.5969744Z STAGE:2023-01-11 21:17:26 4751:4751 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.5970085Z STAGE:2023-01-11 21:17:26 4752:4752 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.5970628Z STAGE:2023-01-11 21:17:26 4751:4751 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.5970968Z STAGE:2023-01-11 21:17:26 4752:4752 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.5971287Z STAGE:2023-01-11 21:17:26 4751:4751 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.5971611Z STAGE:2023-01-11 21:17:26 4752:4752 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.5971937Z STAGE:2023-01-11 21:17:26 4751:4751 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.5972274Z STAGE:2023-01-11 21:17:26 4752:4752 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.5972613Z STAGE:2023-01-11 21:17:26 4751:4751 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.5972928Z STAGE:2023-01-11 21:17:26 4752:4752 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.5973245Z STAGE:2023-01-11 21:17:26 4751:4751 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.5973590Z STAGE:2023-01-11 21:17:26 4752:4752 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.5973911Z STAGE:2023-01-11 21:17:26 4751:4751 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.5974244Z STAGE:2023-01-11 21:17:26 4752:4752 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.5974581Z STAGE:2023-01-11 21:17:26 4751:4751 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.5974772Z ok (4.227s) 2023-01-11T21:44:22.5974792Z 2023-01-11T21:44:22.5975065Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5975177Z Ran 1 test in 4.227s 2023-01-11T21:44:22.5975197Z 2023-01-11T21:44:22.5975288Z OK 2023-01-11T21:44:22.5975308Z 2023-01-11T21:44:22.5975430Z Generating XML reports... 2023-01-11T21:44:22.5975862Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211723.xml 2023-01-11T21:44:22.5976231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5976404Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5976782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5976970Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5976989Z 2023-01-11T21:44:22.5977099Z Running tests... 2023-01-11T21:44:22.5977364Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5977673Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.5977923Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.5978138Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4870 2023-01-11T21:44:22.5978407Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4871 2023-01-11T21:44:22.5978785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5978955Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5979329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5979522Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5979880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5980049Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5980398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5980585Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5980827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.5981066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.5981463Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5981851Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5982079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.5982304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.5982444Z skip: Skipped due to small world size. (4.251s) 2023-01-11T21:44:22.5982481Z 2023-01-11T21:44:22.5982734Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5982845Z Ran 1 test in 4.251s 2023-01-11T21:44:22.5982864Z 2023-01-11T21:44:22.5982971Z OK (skipped=1) 2023-01-11T21:44:22.5982990Z 2023-01-11T21:44:22.5983112Z Generating XML reports... 2023-01-11T21:44:22.5983561Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211729.xml 2023-01-11T21:44:22.5983926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5984153Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5984530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5984700Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5984736Z 2023-01-11T21:44:22.5984826Z Running tests... 2023-01-11T21:44:22.5985094Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5985403Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.5985672Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.5985885Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4979 2023-01-11T21:44:22.5986095Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4980 2023-01-11T21:44:22.5986467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5986637Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5986995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5987181Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5987589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5987763Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5988137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5988322Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5988567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.5988805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.5989182Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5989577Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.5989801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.5990024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.5990350Z STAGE:2023-01-11 21:17:40 4980:4980 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.5990673Z STAGE:2023-01-11 21:17:40 4979:4979 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.5991407Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2588: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T21:44:22.5991516Z warnings.warn( 2023-01-11T21:44:22.5992240Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2588: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T21:44:22.5992349Z warnings.warn( 2023-01-11T21:44:22.5992661Z STAGE:2023-01-11 21:17:40 4979:4979 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.5992983Z STAGE:2023-01-11 21:17:40 4980:4980 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.5993321Z STAGE:2023-01-11 21:17:40 4979:4979 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.5993722Z STAGE:2023-01-11 21:17:40 4980:4980 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.5994041Z STAGE:2023-01-11 21:17:40 4979:4979 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.5994356Z STAGE:2023-01-11 21:17:40 4980:4980 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.5994684Z STAGE:2023-01-11 21:17:40 4979:4979 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.5995010Z STAGE:2023-01-11 21:17:40 4980:4980 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.5995345Z STAGE:2023-01-11 21:17:40 4979:4979 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.5995663Z STAGE:2023-01-11 21:17:40 4980:4980 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.5995986Z STAGE:2023-01-11 21:17:40 4979:4979 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.5996299Z STAGE:2023-01-11 21:17:40 4980:4980 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.5996617Z STAGE:2023-01-11 21:17:40 4979:4979 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.5996935Z STAGE:2023-01-11 21:17:40 4980:4980 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.5997315Z STAGE:2023-01-11 21:17:40 4979:4979 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.5997656Z STAGE:2023-01-11 21:17:40 4980:4980 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.5997756Z ok (4.262s) 2023-01-11T21:44:22.5997776Z 2023-01-11T21:44:22.5998040Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.5998134Z Ran 1 test in 4.262s 2023-01-11T21:44:22.5998153Z 2023-01-11T21:44:22.5998249Z OK 2023-01-11T21:44:22.5998268Z 2023-01-11T21:44:22.5998390Z Generating XML reports... 2023-01-11T21:44:22.5998840Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211736.xml 2023-01-11T21:44:22.5999207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.5999378Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.5999757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.5999945Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.5999965Z 2023-01-11T21:44:22.6000054Z Running tests... 2023-01-11T21:44:22.6000319Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6000629Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6000907Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6001124Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5092 2023-01-11T21:44:22.6001334Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5093 2023-01-11T21:44:22.6001702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6001875Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6002249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6002419Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6002783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6003009Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6003378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6003563Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6003806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6004049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6004441Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6004814Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6005038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6005265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6005594Z STAGE:2023-01-11 21:17:47 5092:5092 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6005915Z STAGE:2023-01-11 21:17:47 5093:5093 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6006814Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2588: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T21:44:22.6006940Z warnings.warn( 2023-01-11T21:44:22.6007725Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2588: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T21:44:22.6007841Z warnings.warn( 2023-01-11T21:44:22.6008378Z STAGE:2023-01-11 21:17:47 5092:5092 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:17:47 5093:5093 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6008398Z 2023-01-11T21:44:22.6008956Z STAGE:2023-01-11 21:17:47 5092:5092 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:17:47 5093:5093 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6008976Z 2023-01-11T21:44:22.6009059Z ok (4.158s) 2023-01-11T21:44:22.6009096Z 2023-01-11T21:44:22.6009343Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6009454Z Ran 1 test in 4.158s 2023-01-11T21:44:22.6009474Z 2023-01-11T21:44:22.6009565Z OK 2023-01-11T21:44:22.6009584Z 2023-01-11T21:44:22.6009705Z Generating XML reports... 2023-01-11T21:44:22.6010151Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211743.xml 2023-01-11T21:44:22.6010883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6011061Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6011438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6011613Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6011647Z 2023-01-11T21:44:22.6011738Z Running tests... 2023-01-11T21:44:22.6012002Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6012311Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6012566Z test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6012871Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5205 2023-01-11T21:44:22.6013083Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5206 2023-01-11T21:44:22.6013453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6013608Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6013987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6014175Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6014535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6014706Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6015076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6015261Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6015505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6015902Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6016185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6016585Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6016812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6017034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6017363Z STAGE:2023-01-11 21:17:54 5205:5205 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6017692Z STAGE:2023-01-11 21:17:54 5206:5206 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6018224Z STAGE:2023-01-11 21:17:54 5206:5206 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:17:54 5205:5205 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6018245Z 2023-01-11T21:44:22.6018800Z STAGE:2023-01-11 21:17:54 5206:5206 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:17:54 5205:5205 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6018820Z 2023-01-11T21:44:22.6019139Z STAGE:2023-01-11 21:17:54 5206:5206 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6019453Z STAGE:2023-01-11 21:17:54 5205:5205 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6019779Z STAGE:2023-01-11 21:17:54 5206:5206 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6020085Z STAGE:2023-01-11 21:17:54 5205:5205 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6020421Z STAGE:2023-01-11 21:17:54 5206:5206 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6020759Z STAGE:2023-01-11 21:17:54 5205:5205 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6020860Z ok (4.230s) 2023-01-11T21:44:22.6020882Z 2023-01-11T21:44:22.6021146Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6021257Z Ran 1 test in 4.231s 2023-01-11T21:44:22.6021276Z 2023-01-11T21:44:22.6021367Z OK 2023-01-11T21:44:22.6021386Z 2023-01-11T21:44:22.6021508Z Generating XML reports... 2023-01-11T21:44:22.6021940Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211750.xml 2023-01-11T21:44:22.6022375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6022546Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6022921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6023109Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6023128Z 2023-01-11T21:44:22.6023239Z Running tests... 2023-01-11T21:44:22.6023500Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6023810Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6024069Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s) 2023-01-11T21:44:22.6024089Z 2023-01-11T21:44:22.6024329Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6024443Z Ran 1 test in 0.002s 2023-01-11T21:44:22.6024462Z 2023-01-11T21:44:22.6024568Z OK (skipped=1) 2023-01-11T21:44:22.6024587Z 2023-01-11T21:44:22.6024709Z Generating XML reports... 2023-01-11T21:44:22.6025153Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211756.xml 2023-01-11T21:44:22.6025520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6025741Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6026124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6026295Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6026332Z 2023-01-11T21:44:22.6026422Z Running tests... 2023-01-11T21:44:22.6026683Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6026996Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6027263Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s) 2023-01-11T21:44:22.6027282Z 2023-01-11T21:44:22.6027541Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6027650Z Ran 1 test in 0.002s 2023-01-11T21:44:22.6027670Z 2023-01-11T21:44:22.6027778Z OK (skipped=1) 2023-01-11T21:44:22.6027798Z 2023-01-11T21:44:22.6027918Z Generating XML reports... 2023-01-11T21:44:22.6028345Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211759.xml 2023-01-11T21:44:22.6028709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6028879Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6029256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6029443Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6029462Z 2023-01-11T21:44:22.6029569Z Running tests... 2023-01-11T21:44:22.6029831Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6030144Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6030401Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6030600Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5384 2023-01-11T21:44:22.6030811Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5385 2023-01-11T21:44:22.6031173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6031399Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6031773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6031959Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6032319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6032492Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6032842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6033027Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6033267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6033507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6033905Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6034295Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6034520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6034787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6035028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T21:44:22.6035246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T21:44:22.6035642Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.6036033Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.6036357Z STAGE:2023-01-11 21:18:05 5384:5384 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6036677Z STAGE:2023-01-11 21:18:05 5385:5385 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6037007Z STAGE:2023-01-11 21:18:05 5385:5385 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6037327Z STAGE:2023-01-11 21:18:05 5384:5384 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6037667Z STAGE:2023-01-11 21:18:05 5385:5385 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6038003Z STAGE:2023-01-11 21:18:05 5384:5384 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6038300Z STAGE:2023-01-11 21:18:05 5385:5385 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6038621Z STAGE:2023-01-11 21:18:05 5384:5384 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6038944Z STAGE:2023-01-11 21:18:05 5385:5385 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6039269Z STAGE:2023-01-11 21:18:05 5384:5384 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6039608Z STAGE:2023-01-11 21:18:05 5385:5385 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6039944Z STAGE:2023-01-11 21:18:05 5384:5384 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6040045Z ok (4.274s) 2023-01-11T21:44:22.6040065Z 2023-01-11T21:44:22.6040329Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6040439Z Ran 1 test in 4.274s 2023-01-11T21:44:22.6040459Z 2023-01-11T21:44:22.6040532Z OK 2023-01-11T21:44:22.6040550Z 2023-01-11T21:44:22.6040724Z Generating XML reports... 2023-01-11T21:44:22.6041175Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211801.xml 2023-01-11T21:44:22.6041543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6041715Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6042091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6042281Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6042301Z 2023-01-11T21:44:22.6042408Z Running tests... 2023-01-11T21:44:22.6042655Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6042967Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6043221Z test_all_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6043439Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5503 2023-01-11T21:44:22.6043651Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5504 2023-01-11T21:44:22.6044016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6044236Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6044620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6044807Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6045150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6045321Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6045691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6045875Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6046117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6046357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6046754Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6047144Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6047368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6047573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6047732Z skip: Skipped due to small world size. (4.202s) 2023-01-11T21:44:22.6047752Z 2023-01-11T21:44:22.6048014Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6048126Z Ran 1 test in 4.202s 2023-01-11T21:44:22.6048145Z 2023-01-11T21:44:22.6048249Z OK (skipped=1) 2023-01-11T21:44:22.6048268Z 2023-01-11T21:44:22.6048389Z Generating XML reports... 2023-01-11T21:44:22.6048841Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211808.xml 2023-01-11T21:44:22.6049206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6049361Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6049734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6049979Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6049998Z 2023-01-11T21:44:22.6050106Z Running tests... 2023-01-11T21:44:22.6050545Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6050865Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6051163Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s) 2023-01-11T21:44:22.6051184Z 2023-01-11T21:44:22.6051451Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6051562Z Ran 1 test in 0.002s 2023-01-11T21:44:22.6051582Z 2023-01-11T21:44:22.6051671Z OK (skipped=1) 2023-01-11T21:44:22.6051690Z 2023-01-11T21:44:22.6051811Z Generating XML reports... 2023-01-11T21:44:22.6052254Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211815.xml 2023-01-11T21:44:22.6052619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6052791Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6053164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6053352Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6053371Z 2023-01-11T21:44:22.6053550Z Running tests... 2023-01-11T21:44:22.6053824Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6054118Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6054411Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s) 2023-01-11T21:44:22.6054430Z 2023-01-11T21:44:22.6054686Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6054799Z Ran 1 test in 0.002s 2023-01-11T21:44:22.6054818Z 2023-01-11T21:44:22.6054924Z OK (skipped=1) 2023-01-11T21:44:22.6054943Z 2023-01-11T21:44:22.6055064Z Generating XML reports... 2023-01-11T21:44:22.6055506Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211817.xml 2023-01-11T21:44:22.6055873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6056044Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6056399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6056587Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6056606Z 2023-01-11T21:44:22.6056713Z Running tests... 2023-01-11T21:44:22.6056977Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6057285Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6057565Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s) 2023-01-11T21:44:22.6057585Z 2023-01-11T21:44:22.6057844Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6057957Z Ran 1 test in 0.002s 2023-01-11T21:44:22.6057976Z 2023-01-11T21:44:22.6058065Z OK (skipped=1) 2023-01-11T21:44:22.6058100Z 2023-01-11T21:44:22.6058204Z Generating XML reports... 2023-01-11T21:44:22.6058642Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211820.xml 2023-01-11T21:44:22.6059006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6059259Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6059634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6059818Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6059837Z 2023-01-11T21:44:22.6059943Z Running tests... 2023-01-11T21:44:22.6060205Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6060495Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6060787Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s) 2023-01-11T21:44:22.6060807Z 2023-01-11T21:44:22.6061067Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6061177Z Ran 1 test in 0.002s 2023-01-11T21:44:22.6061196Z 2023-01-11T21:44:22.6061301Z OK (skipped=1) 2023-01-11T21:44:22.6061323Z 2023-01-11T21:44:22.6061445Z Generating XML reports... 2023-01-11T21:44:22.6061885Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211822.xml 2023-01-11T21:44:22.6062250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6062421Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6062823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6063015Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6063034Z 2023-01-11T21:44:22.6063141Z Running tests... 2023-01-11T21:44:22.6063403Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6063710Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6063982Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6064196Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5744 2023-01-11T21:44:22.6064408Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5745 2023-01-11T21:44:22.6064756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6064933Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6065305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6065489Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6065848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6066020Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6066393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6066577Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6066817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6067043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6067438Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6067826Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6068049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6068324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6068426Z ok (4.213s) 2023-01-11T21:44:22.6068446Z 2023-01-11T21:44:22.6068715Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6068825Z Ran 1 test in 4.213s 2023-01-11T21:44:22.6068845Z 2023-01-11T21:44:22.6068918Z OK 2023-01-11T21:44:22.6068953Z 2023-01-11T21:44:22.6069059Z Generating XML reports... 2023-01-11T21:44:22.6069506Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211824.xml 2023-01-11T21:44:22.6069874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6070045Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6070413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6070602Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6070621Z 2023-01-11T21:44:22.6070728Z Running tests... 2023-01-11T21:44:22.6070988Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6071279Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6071545Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6071804Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5853 2023-01-11T21:44:22.6072019Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5854 2023-01-11T21:44:22.6072386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6072556Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6072927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6073117Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6073517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6073671Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6074042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6074225Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6074467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6074708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6075102Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6075499Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6075723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6075927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6076165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T21:44:22.6076400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T21:44:22.6076791Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.6077173Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.6077460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2023-01-11T21:44:22.6077691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2023-01-11T21:44:22.6078083Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2023-01-11T21:44:22.6078471Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2023-01-11T21:44:22.6078705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2023-01-11T21:44:22.6078919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2023-01-11T21:44:22.6079302Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2023-01-11T21:44:22.6079687Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2023-01-11T21:44:22.6079792Z ok (4.312s) 2023-01-11T21:44:22.6079812Z 2023-01-11T21:44:22.6080075Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6080185Z Ran 1 test in 4.312s 2023-01-11T21:44:22.6080205Z 2023-01-11T21:44:22.6080297Z OK 2023-01-11T21:44:22.6080316Z 2023-01-11T21:44:22.6080439Z Generating XML reports... 2023-01-11T21:44:22.6080913Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211831.xml 2023-01-11T21:44:22.6081288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6081460Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6081832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6082024Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6082043Z 2023-01-11T21:44:22.6082150Z Running tests... 2023-01-11T21:44:22.6082411Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6082720Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6082978Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports all_gather_v (0.002s) 2023-01-11T21:44:22.6082998Z 2023-01-11T21:44:22.6083240Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6083350Z Ran 1 test in 0.003s 2023-01-11T21:44:22.6083368Z 2023-01-11T21:44:22.6083475Z OK (skipped=1) 2023-01-11T21:44:22.6083494Z 2023-01-11T21:44:22.6083615Z Generating XML reports... 2023-01-11T21:44:22.6084055Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211838.xml 2023-01-11T21:44:22.6084426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6084598Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6084971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6085156Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6085179Z 2023-01-11T21:44:22.6085268Z Running tests... 2023-01-11T21:44:22.6085531Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6085841Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6086120Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6086334Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6019 2023-01-11T21:44:22.6086620Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6020 2023-01-11T21:44:22.6086986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6087159Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6087513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6087704Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6088060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6088231Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6088597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6088784Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6089024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6089262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6089652Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6090066Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6090460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6090695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6090929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T21:44:22.6091171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T21:44:22.6091571Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.6091953Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.6092283Z STAGE:2023-01-11 21:18:44 6019:6019 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6092606Z STAGE:2023-01-11 21:18:44 6020:6020 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6093319Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T21:44:22.6093438Z warnings.warn( 2023-01-11T21:44:22.6094158Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T21:44:22.6094265Z warnings.warn( 2023-01-11T21:44:22.6094595Z STAGE:2023-01-11 21:18:44 6019:6019 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6094921Z STAGE:2023-01-11 21:18:44 6020:6020 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6095259Z STAGE:2023-01-11 21:18:44 6019:6019 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6095597Z STAGE:2023-01-11 21:18:44 6020:6020 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6095917Z STAGE:2023-01-11 21:18:44 6019:6019 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6096324Z STAGE:2023-01-11 21:18:44 6020:6020 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6096634Z STAGE:2023-01-11 21:18:44 6019:6019 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6096951Z STAGE:2023-01-11 21:18:44 6020:6020 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6097292Z STAGE:2023-01-11 21:18:44 6019:6019 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6097626Z STAGE:2023-01-11 21:18:44 6020:6020 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6097727Z ok (4.215s) 2023-01-11T21:44:22.6097747Z 2023-01-11T21:44:22.6098011Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6098121Z Ran 1 test in 4.215s 2023-01-11T21:44:22.6098140Z 2023-01-11T21:44:22.6098232Z OK 2023-01-11T21:44:22.6098251Z 2023-01-11T21:44:22.6098355Z Generating XML reports... 2023-01-11T21:44:22.6098807Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211840.xml 2023-01-11T21:44:22.6099174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6099346Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6099777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6099971Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6099991Z 2023-01-11T21:44:22.6100097Z Running tests... 2023-01-11T21:44:22.6100362Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6100674Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6100935Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6101153Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6138 2023-01-11T21:44:22.6101364Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6139 2023-01-11T21:44:22.6101733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6101907Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6102279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6102465Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6102824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6102975Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6103343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6103527Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6103768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6104006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6104403Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6104793Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6105016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6105237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6105518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T21:44:22.6105754Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T21:44:22.6106146Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.6106542Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.6106874Z STAGE:2023-01-11 21:18:51 6138:6138 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6107181Z STAGE:2023-01-11 21:18:51 6139:6139 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6107911Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T21:44:22.6108027Z warnings.warn( 2023-01-11T21:44:22.6108747Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T21:44:22.6108900Z warnings.warn( 2023-01-11T21:44:22.6109220Z STAGE:2023-01-11 21:18:51 6138:6138 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6109544Z STAGE:2023-01-11 21:18:51 6139:6139 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6109882Z STAGE:2023-01-11 21:18:51 6138:6138 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6110217Z STAGE:2023-01-11 21:18:51 6139:6139 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6110537Z STAGE:2023-01-11 21:18:51 6138:6138 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6110853Z STAGE:2023-01-11 21:18:51 6139:6139 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6111383Z STAGE:2023-01-11 21:18:51 6138:6138 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:18:51 6139:6139 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6111406Z 2023-01-11T21:44:22.6111739Z STAGE:2023-01-11 21:18:51 6138:6138 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6112070Z STAGE:2023-01-11 21:18:51 6139:6139 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6112170Z ok (4.239s) 2023-01-11T21:44:22.6112190Z 2023-01-11T21:44:22.6112436Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6112551Z Ran 1 test in 4.240s 2023-01-11T21:44:22.6112570Z 2023-01-11T21:44:22.6112661Z OK 2023-01-11T21:44:22.6112680Z 2023-01-11T21:44:22.6112802Z Generating XML reports... 2023-01-11T21:44:22.6113247Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211847.xml 2023-01-11T21:44:22.6113613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6113788Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6114161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6114330Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6114367Z 2023-01-11T21:44:22.6114457Z Running tests... 2023-01-11T21:44:22.6114719Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6115030Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6115370Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6115582Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6257 2023-01-11T21:44:22.6115794Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6258 2023-01-11T21:44:22.6116164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6116335Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6116689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6116874Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6117229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6117414Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6117777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6117967Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6118244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6118511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6118867Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6119260Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6119487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6119714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6119948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T21:44:22.6120184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T21:44:22.6120580Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.6120965Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.6121289Z STAGE:2023-01-11 21:18:58 6258:6258 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6121591Z STAGE:2023-01-11 21:18:58 6257:6257 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6122326Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T21:44:22.6122441Z warnings.warn( 2023-01-11T21:44:22.6123168Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T21:44:22.6123278Z warnings.warn( 2023-01-11T21:44:22.6123814Z STAGE:2023-01-11 21:18:58 6257:6257 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:18:58 6258:6258 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6123834Z 2023-01-11T21:44:22.6124385Z STAGE:2023-01-11 21:18:58 6257:6257 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:18:58 6258:6258 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6124452Z 2023-01-11T21:44:22.6124777Z STAGE:2023-01-11 21:18:58 6257:6257 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6125093Z STAGE:2023-01-11 21:18:58 6258:6258 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6125421Z STAGE:2023-01-11 21:18:58 6257:6257 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6125740Z STAGE:2023-01-11 21:18:58 6258:6258 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6126076Z STAGE:2023-01-11 21:18:58 6257:6257 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6126395Z STAGE:2023-01-11 21:18:58 6258:6258 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6126496Z ok (4.298s) 2023-01-11T21:44:22.6126518Z 2023-01-11T21:44:22.6126781Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6126890Z Ran 1 test in 4.298s 2023-01-11T21:44:22.6126909Z 2023-01-11T21:44:22.6126999Z OK 2023-01-11T21:44:22.6127019Z 2023-01-11T21:44:22.6127140Z Generating XML reports... 2023-01-11T21:44:22.6127589Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211854.xml 2023-01-11T21:44:22.6128002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6128163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6128540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6128729Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6128749Z 2023-01-11T21:44:22.6128859Z Running tests... 2023-01-11T21:44:22.6129124Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6129434Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6129712Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6129926Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6376 2023-01-11T21:44:22.6130142Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6377 2023-01-11T21:44:22.6130667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6130845Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6131219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6131411Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6131770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6131938Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6132307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6132497Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6132720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6132958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6133353Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6133744Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6134053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6134276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6134509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T21:44:22.6134748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T21:44:22.6135145Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.6135513Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.6135840Z STAGE:2023-01-11 21:19:05 6377:6377 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6136167Z STAGE:2023-01-11 21:19:05 6376:6376 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6136898Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T21:44:22.6137010Z warnings.warn( 2023-01-11T21:44:22.6137788Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T21:44:22.6137904Z warnings.warn( 2023-01-11T21:44:22.6138442Z STAGE:2023-01-11 21:19:05 6376:6376 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:19:05 6377:6377 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6138465Z 2023-01-11T21:44:22.6139020Z STAGE:2023-01-11 21:19:05 6377:6377 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:19:05 6376:6376 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6139040Z 2023-01-11T21:44:22.6139356Z STAGE:2023-01-11 21:19:05 6376:6376 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6139675Z STAGE:2023-01-11 21:19:05 6377:6377 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6140002Z STAGE:2023-01-11 21:19:05 6376:6376 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6140311Z STAGE:2023-01-11 21:19:05 6377:6377 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6140648Z STAGE:2023-01-11 21:19:05 6376:6376 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6140988Z STAGE:2023-01-11 21:19:05 6377:6377 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6141090Z ok (4.245s) 2023-01-11T21:44:22.6141109Z 2023-01-11T21:44:22.6141374Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6141485Z Ran 1 test in 4.245s 2023-01-11T21:44:22.6141504Z 2023-01-11T21:44:22.6141596Z OK 2023-01-11T21:44:22.6141615Z 2023-01-11T21:44:22.6141739Z Generating XML reports... 2023-01-11T21:44:22.6142173Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211901.xml 2023-01-11T21:44:22.6142541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6142714Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6143088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6143330Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6143349Z 2023-01-11T21:44:22.6143455Z Running tests... 2023-01-11T21:44:22.6143721Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6144030Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6144304Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6144501Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6495 2023-01-11T21:44:22.6144710Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6496 2023-01-11T21:44:22.6145075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6145247Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6145629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6145817Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6146177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6146348Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6146751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6146940Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6147180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6147416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6147808Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6148203Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6148426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6148647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6148806Z skip: Skipped due to small world size. (4.263s) 2023-01-11T21:44:22.6148827Z 2023-01-11T21:44:22.6149071Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6149183Z Ran 1 test in 4.263s 2023-01-11T21:44:22.6149202Z 2023-01-11T21:44:22.6149308Z OK (skipped=1) 2023-01-11T21:44:22.6149327Z 2023-01-11T21:44:22.6149449Z Generating XML reports... 2023-01-11T21:44:22.6149893Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211907.xml 2023-01-11T21:44:22.6150257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6150428Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6150798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6150988Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6151007Z 2023-01-11T21:44:22.6151097Z Running tests... 2023-01-11T21:44:22.6151363Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6151671Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6151940Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6152152Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6604 2023-01-11T21:44:22.6152429Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6605 2023-01-11T21:44:22.6152797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6152969Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6153330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6153517Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6153877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6154043Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6154409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6154594Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6154829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6155063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6155455Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6155886Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6156116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6156339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6156492Z skip: Skipped due to small world size. (4.144s) 2023-01-11T21:44:22.6156515Z 2023-01-11T21:44:22.6156777Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6156886Z Ran 1 test in 4.144s 2023-01-11T21:44:22.6156906Z 2023-01-11T21:44:22.6157008Z OK (skipped=1) 2023-01-11T21:44:22.6157027Z 2023-01-11T21:44:22.6157146Z Generating XML reports... 2023-01-11T21:44:22.6157591Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211914.xml 2023-01-11T21:44:22.6157943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6158114Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6158483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6158665Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6158684Z 2023-01-11T21:44:22.6158788Z Running tests... 2023-01-11T21:44:22.6159050Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6159350Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6159628Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6159824Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6713 2023-01-11T21:44:22.6160037Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6714 2023-01-11T21:44:22.6160399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6160568Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6160939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6161123Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6161536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6161704Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6162063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6162234Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6162471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6162709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6163101Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6163486Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6163713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6163934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6164089Z skip: Skipped due to small world size. (4.225s) 2023-01-11T21:44:22.6164108Z 2023-01-11T21:44:22.6164413Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6164514Z Ran 1 test in 4.225s 2023-01-11T21:44:22.6164533Z 2023-01-11T21:44:22.6164634Z OK (skipped=1) 2023-01-11T21:44:22.6164653Z 2023-01-11T21:44:22.6164768Z Generating XML reports... 2023-01-11T21:44:22.6165210Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211921.xml 2023-01-11T21:44:22.6165564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6165732Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6166097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6166274Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6166294Z 2023-01-11T21:44:22.6166384Z Running tests... 2023-01-11T21:44:22.6166642Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6166943Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6167209Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6167423Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6822 2023-01-11T21:44:22.6167625Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6823 2023-01-11T21:44:22.6167989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6168152Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6168518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6168687Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6169038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6169199Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6169552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6169726Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6169954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6170240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6170794Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6171173Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6171391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6171604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6171749Z skip: Skipped due to small world size. (4.195s) 2023-01-11T21:44:22.6171769Z 2023-01-11T21:44:22.6172022Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6172127Z Ran 1 test in 4.195s 2023-01-11T21:44:22.6172150Z 2023-01-11T21:44:22.6172255Z OK (skipped=1) 2023-01-11T21:44:22.6172274Z 2023-01-11T21:44:22.6172387Z Generating XML reports... 2023-01-11T21:44:22.6172824Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211928.xml 2023-01-11T21:44:22.6173170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6173443Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6173829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6174006Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6174026Z 2023-01-11T21:44:22.6174122Z Running tests... 2023-01-11T21:44:22.6174374Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6174672Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6174931Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6175135Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6931 2023-01-11T21:44:22.6175328Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6932 2023-01-11T21:44:22.6175686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6175846Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6176212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6176392Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6176737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6176899Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6177261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6177429Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6177658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6177889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6178274Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6178653Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6178868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6179156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6179477Z STAGE:2023-01-11 21:19:38 6932:6932 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6179791Z STAGE:2023-01-11 21:19:38 6931:6931 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6180507Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T21:44:22.6180619Z warnings.warn( 2023-01-11T21:44:22.6181339Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T21:44:22.6181450Z warnings.warn( 2023-01-11T21:44:22.6181779Z STAGE:2023-01-11 21:19:38 6931:6931 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6182096Z STAGE:2023-01-11 21:19:38 6932:6932 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6182691Z STAGE:2023-01-11 21:19:38 6931:6931 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:19:38 6932:6932 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6182712Z 2023-01-11T21:44:22.6183228Z STAGE:2023-01-11 21:19:38 6931:6931 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:19:38 6932:6932 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6183247Z 2023-01-11T21:44:22.6183568Z STAGE:2023-01-11 21:19:38 6931:6931 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6183889Z STAGE:2023-01-11 21:19:38 6932:6932 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6184226Z STAGE:2023-01-11 21:19:38 6931:6931 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6184544Z STAGE:2023-01-11 21:19:38 6932:6932 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6184644Z ok (4.215s) 2023-01-11T21:44:22.6184663Z 2023-01-11T21:44:22.6184928Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6185040Z Ran 1 test in 4.215s 2023-01-11T21:44:22.6185059Z 2023-01-11T21:44:22.6185150Z OK 2023-01-11T21:44:22.6185169Z 2023-01-11T21:44:22.6185283Z Generating XML reports... 2023-01-11T21:44:22.6185721Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211934.xml 2023-01-11T21:44:22.6186080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6186247Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6186605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6186785Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6186805Z 2023-01-11T21:44:22.6186901Z Running tests... 2023-01-11T21:44:22.6187157Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6187458Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6187748Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6187961Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7044 2023-01-11T21:44:22.6188169Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7045 2023-01-11T21:44:22.6188574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6188746Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6189121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6189308Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6189668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6189837Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6190193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6190375Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6190611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6190834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6191221Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6191645Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6191868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6192090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6192822Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T21:44:22.6192938Z warnings.warn( 2023-01-11T21:44:22.6193657Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T21:44:22.6193755Z warnings.warn( 2023-01-11T21:44:22.6193837Z ok (4.220s) 2023-01-11T21:44:22.6193868Z 2023-01-11T21:44:22.6194118Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6194220Z Ran 1 test in 4.220s 2023-01-11T21:44:22.6194239Z 2023-01-11T21:44:22.6194320Z OK 2023-01-11T21:44:22.6194339Z 2023-01-11T21:44:22.6194450Z Generating XML reports... 2023-01-11T21:44:22.6194887Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211941.xml 2023-01-11T21:44:22.6195247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6195410Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6195773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6195942Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6195968Z 2023-01-11T21:44:22.6196062Z Running tests... 2023-01-11T21:44:22.6196314Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6196613Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6196867Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6197070Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7153 2023-01-11T21:44:22.6197330Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7154 2023-01-11T21:44:22.6197686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6197842Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6198205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6198384Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6198734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6198895Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6199256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6199431Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6199667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6199898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6200274Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6200704Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6200927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6201141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6201466Z STAGE:2023-01-11 21:19:52 7154:7154 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6201779Z STAGE:2023-01-11 21:19:52 7153:7153 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6202502Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T21:44:22.6202603Z warnings.warn( 2023-01-11T21:44:22.6203317Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T21:44:22.6203416Z warnings.warn( 2023-01-11T21:44:22.6203731Z STAGE:2023-01-11 21:19:52 7154:7154 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6204044Z STAGE:2023-01-11 21:19:52 7153:7153 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6204380Z STAGE:2023-01-11 21:19:52 7154:7154 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6204709Z STAGE:2023-01-11 21:19:52 7153:7153 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6205215Z STAGE:2023-01-11 21:19:52 7154:7154 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:19:52 7153:7153 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6205238Z 2023-01-11T21:44:22.6205550Z STAGE:2023-01-11 21:19:52 7154:7154 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6205865Z STAGE:2023-01-11 21:19:52 7153:7153 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6206192Z STAGE:2023-01-11 21:19:52 7154:7154 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6206516Z STAGE:2023-01-11 21:19:52 7153:7153 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6206651Z ok (4.229s) 2023-01-11T21:44:22.6206679Z 2023-01-11T21:44:22.6206928Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6207031Z Ran 1 test in 4.229s 2023-01-11T21:44:22.6207050Z 2023-01-11T21:44:22.6207132Z OK 2023-01-11T21:44:22.6207151Z 2023-01-11T21:44:22.6207266Z Generating XML reports... 2023-01-11T21:44:22.6207710Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211948.xml 2023-01-11T21:44:22.6208071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6208236Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6208601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6208772Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6208795Z 2023-01-11T21:44:22.6208894Z Running tests... 2023-01-11T21:44:22.6209150Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6209448Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6209713Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6209966Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7266 2023-01-11T21:44:22.6210174Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7267 2023-01-11T21:44:22.6210796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6210954Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6211324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6211508Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6211859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6212020Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6212383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6212558Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6212788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6213173Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6213387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6213775Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6213988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6214199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6214524Z STAGE:2023-01-11 21:19:59 7266:7266 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6214842Z STAGE:2023-01-11 21:19:59 7267:7267 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6215560Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T21:44:22.6215759Z warnings.warn( 2023-01-11T21:44:22.6216481Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T21:44:22.6216583Z warnings.warn( 2023-01-11T21:44:22.6217105Z STAGE:2023-01-11 21:19:59 7267:7267 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:19:59 7266:7266 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6217138Z 2023-01-11T21:44:22.6217673Z STAGE:2023-01-11 21:19:59 7267:7267 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:19:59 7266:7266 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6217701Z 2023-01-11T21:44:22.6218002Z STAGE:2023-01-11 21:19:59 7267:7267 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6218316Z STAGE:2023-01-11 21:19:59 7266:7266 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6218634Z STAGE:2023-01-11 21:19:59 7267:7267 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6218956Z STAGE:2023-01-11 21:19:59 7266:7266 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6219341Z STAGE:2023-01-11 21:19:59 7267:7267 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6219685Z STAGE:2023-01-11 21:19:59 7266:7266 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6219778Z ok (4.246s) 2023-01-11T21:44:22.6219797Z 2023-01-11T21:44:22.6220063Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6220157Z Ran 1 test in 4.246s 2023-01-11T21:44:22.6220177Z 2023-01-11T21:44:22.6220260Z OK 2023-01-11T21:44:22.6220279Z 2023-01-11T21:44:22.6220402Z Generating XML reports... 2023-01-11T21:44:22.6220850Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211955.xml 2023-01-11T21:44:22.6221214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6221380Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6221755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6221943Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6221962Z 2023-01-11T21:44:22.6222060Z Running tests... 2023-01-11T21:44:22.6222308Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6222616Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6222878Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6223095Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7379 2023-01-11T21:44:22.6223302Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7380 2023-01-11T21:44:22.6223656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6223821Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6224186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6224357Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6224706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6224865Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6225336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6225509Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6225741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6225970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6226356Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6226740Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6226949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6227163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6227484Z STAGE:2023-01-11 21:20:05 7380:7380 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6227801Z STAGE:2023-01-11 21:20:05 7379:7379 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6228563Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T21:44:22.6228676Z warnings.warn( 2023-01-11T21:44:22.6229392Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T21:44:22.6229493Z warnings.warn( 2023-01-11T21:44:22.6229820Z STAGE:2023-01-11 21:20:05 7380:7380 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6230123Z STAGE:2023-01-11 21:20:05 7379:7379 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6230453Z STAGE:2023-01-11 21:20:05 7380:7380 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6230778Z STAGE:2023-01-11 21:20:05 7379:7379 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6231095Z STAGE:2023-01-11 21:20:05 7380:7380 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6231405Z STAGE:2023-01-11 21:20:05 7379:7379 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6231719Z STAGE:2023-01-11 21:20:05 7380:7380 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6232035Z STAGE:2023-01-11 21:20:05 7379:7379 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6232365Z STAGE:2023-01-11 21:20:05 7380:7380 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6232688Z STAGE:2023-01-11 21:20:05 7379:7379 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6232772Z ok (4.250s) 2023-01-11T21:44:22.6232791Z 2023-01-11T21:44:22.6233047Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6233149Z Ran 1 test in 4.250s 2023-01-11T21:44:22.6233181Z 2023-01-11T21:44:22.6233268Z OK 2023-01-11T21:44:22.6233275Z 2023-01-11T21:44:22.6233390Z Generating XML reports... 2023-01-11T21:44:22.6233826Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212002.xml 2023-01-11T21:44:22.6234181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6234347Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6234791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6234962Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6234981Z 2023-01-11T21:44:22.6235080Z Running tests... 2023-01-11T21:44:22.6235336Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6235642Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6235912Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6236119Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7492 2023-01-11T21:44:22.6236319Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7493 2023-01-11T21:44:22.6236674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6236832Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6237195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6237372Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6237724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6237931Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6238310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6238486Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6238726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6238956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6239338Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6239727Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6239944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6240168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6240263Z ok (4.159s) 2023-01-11T21:44:22.6240283Z 2023-01-11T21:44:22.6240545Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6240647Z Ran 1 test in 4.159s 2023-01-11T21:44:22.6240666Z 2023-01-11T21:44:22.6240757Z OK 2023-01-11T21:44:22.6240775Z 2023-01-11T21:44:22.6240879Z Generating XML reports... 2023-01-11T21:44:22.6241325Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212008.xml 2023-01-11T21:44:22.6241690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6241853Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6242225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6242411Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6242430Z 2023-01-11T21:44:22.6242536Z Running tests... 2023-01-11T21:44:22.6242802Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6243105Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6243351Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6243619Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7601 2023-01-11T21:44:22.6243828Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7602 2023-01-11T21:44:22.6244187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6244358Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6244734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6244913Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6245269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6245421Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6245777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6245963Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6246204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6246433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6246873Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6247273Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6247489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6247713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6247933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T21:44:22.6248161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T21:44:22.6248551Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.6248941Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.6249260Z STAGE:2023-01-11 21:20:19 7602:7602 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6249580Z STAGE:2023-01-11 21:20:19 7601:7601 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6250117Z STAGE:2023-01-11 21:20:19 7602:7602 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:20:19 7601:7601 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6250141Z 2023-01-11T21:44:22.6250920Z STAGE:2023-01-11 21:20:19 7602:7602 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:20:19 7601:7601 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6250941Z 2023-01-11T21:44:22.6251268Z STAGE:2023-01-11 21:20:19 7602:7602 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6251581Z STAGE:2023-01-11 21:20:19 7601:7601 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6252111Z STAGE:2023-01-11 21:20:19 7601:7601 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:20:19 7602:7602 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6252131Z 2023-01-11T21:44:22.6252672Z STAGE:2023-01-11 21:20:19 7602:7602 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:20:19 7601:7601 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6252771Z 2023-01-11T21:44:22.6252861Z ok (4.250s) 2023-01-11T21:44:22.6252892Z 2023-01-11T21:44:22.6253143Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6253254Z Ran 1 test in 4.250s 2023-01-11T21:44:22.6253273Z 2023-01-11T21:44:22.6253364Z OK 2023-01-11T21:44:22.6253383Z 2023-01-11T21:44:22.6253496Z Generating XML reports... 2023-01-11T21:44:22.6253946Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212015.xml 2023-01-11T21:44:22.6254312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6254479Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6254851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6255021Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6255043Z 2023-01-11T21:44:22.6255144Z Running tests... 2023-01-11T21:44:22.6255407Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6255708Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6255968Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6256235Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7720 2023-01-11T21:44:22.6256455Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7721 2023-01-11T21:44:22.6256814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6256968Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6257337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6257519Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6257875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6258042Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6258410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6258593Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6258827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6259067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6259444Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6259829Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6260053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6260266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6260503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T21:44:22.6260740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T21:44:22.6261123Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.6261505Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.6261822Z STAGE:2023-01-11 21:20:26 7721:7721 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6262184Z STAGE:2023-01-11 21:20:26 7720:7720 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6262510Z STAGE:2023-01-11 21:20:26 7720:7720 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6262841Z STAGE:2023-01-11 21:20:26 7720:7720 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6263164Z STAGE:2023-01-11 21:20:26 7720:7720 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6263481Z STAGE:2023-01-11 21:20:26 7721:7721 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6263816Z STAGE:2023-01-11 21:20:26 7721:7721 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6264132Z STAGE:2023-01-11 21:20:26 7721:7721 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6264451Z STAGE:2023-01-11 21:20:26 7720:7720 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6264772Z STAGE:2023-01-11 21:20:26 7721:7721 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6265090Z STAGE:2023-01-11 21:20:26 7720:7720 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6265422Z STAGE:2023-01-11 21:20:26 7721:7721 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6265560Z ok (4.241s) 2023-01-11T21:44:22.6265581Z 2023-01-11T21:44:22.6265850Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6265952Z Ran 1 test in 4.241s 2023-01-11T21:44:22.6265971Z 2023-01-11T21:44:22.6266061Z OK 2023-01-11T21:44:22.6266080Z 2023-01-11T21:44:22.6266199Z Generating XML reports... 2023-01-11T21:44:22.6266638Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212022.xml 2023-01-11T21:44:22.6266992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6267164Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6267535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6267721Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6267740Z 2023-01-11T21:44:22.6267841Z Running tests... 2023-01-11T21:44:22.6268102Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6268405Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6268675Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6268879Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7839 2023-01-11T21:44:22.6269077Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7840 2023-01-11T21:44:22.6269442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6269612Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6269974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6270164Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6270524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6270694Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6271054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6271278Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6271519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6271749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6272144Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6272536Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6272760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6272977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6273209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T21:44:22.6273487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T21:44:22.6273863Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.6274243Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.6274613Z STAGE:2023-01-11 21:20:33 7840:7840 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6274944Z STAGE:2023-01-11 21:20:33 7839:7839 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6275470Z STAGE:2023-01-11 21:20:33 7840:7840 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:20:33 7839:7839 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6275490Z 2023-01-11T21:44:22.6276039Z STAGE:2023-01-11 21:20:33 7840:7840 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:20:33 7839:7839 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6276063Z 2023-01-11T21:44:22.6276576Z STAGE:2023-01-11 21:20:33 7839:7839 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:20:33 7840:7840 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6276596Z 2023-01-11T21:44:22.6276919Z STAGE:2023-01-11 21:20:33 7840:7840 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6277241Z STAGE:2023-01-11 21:20:33 7839:7839 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6277575Z STAGE:2023-01-11 21:20:33 7840:7840 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6277906Z STAGE:2023-01-11 21:20:33 7839:7839 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6278006Z ok (4.234s) 2023-01-11T21:44:22.6278026Z 2023-01-11T21:44:22.6278278Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6278389Z Ran 1 test in 4.234s 2023-01-11T21:44:22.6278409Z 2023-01-11T21:44:22.6278492Z OK 2023-01-11T21:44:22.6278510Z 2023-01-11T21:44:22.6278632Z Generating XML reports... 2023-01-11T21:44:22.6279076Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212029.xml 2023-01-11T21:44:22.6279436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6279610Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6279976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6280145Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6280180Z 2023-01-11T21:44:22.6280271Z Running tests... 2023-01-11T21:44:22.6280598Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6280907Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6281164Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6281378Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7958 2023-01-11T21:44:22.6281591Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7959 2023-01-11T21:44:22.6281949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6282120Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6282473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6282659Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6283009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6283176Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6283541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6283717Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6284005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6284242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6284620Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6285009Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6285234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6285447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6285680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T21:44:22.6285909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T21:44:22.6286298Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.6286679Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.6287004Z STAGE:2023-01-11 21:20:39 7959:7959 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6287308Z STAGE:2023-01-11 21:20:39 7958:7958 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6287833Z STAGE:2023-01-11 21:20:39 7958:7958 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:20:39 7959:7959 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6287853Z 2023-01-11T21:44:22.6288408Z STAGE:2023-01-11 21:20:39 7959:7959 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:20:39 7958:7958 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6288428Z 2023-01-11T21:44:22.6288737Z STAGE:2023-01-11 21:20:39 7959:7959 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6289053Z STAGE:2023-01-11 21:20:39 7958:7958 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6289378Z STAGE:2023-01-11 21:20:39 7959:7959 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6289744Z STAGE:2023-01-11 21:20:39 7958:7958 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6290076Z STAGE:2023-01-11 21:20:39 7959:7959 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6290573Z STAGE:2023-01-11 21:20:39 7958:7958 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6290681Z ok (4.257s) 2023-01-11T21:44:22.6290701Z 2023-01-11T21:44:22.6290969Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6291065Z Ran 1 test in 4.257s 2023-01-11T21:44:22.6291086Z 2023-01-11T21:44:22.6291177Z OK 2023-01-11T21:44:22.6291196Z 2023-01-11T21:44:22.6291317Z Generating XML reports... 2023-01-11T21:44:22.6291757Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212035.xml 2023-01-11T21:44:22.6292128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6292297Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6292670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6292850Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6292869Z 2023-01-11T21:44:22.6292960Z Running tests... 2023-01-11T21:44:22.6293292Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6293607Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6293863Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6294078Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8077 2023-01-11T21:44:22.6294280Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8078 2023-01-11T21:44:22.6294652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6294813Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6295185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6295355Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6295716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6295876Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6296239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6296422Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6296654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6296895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6297287Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6297661Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6297879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6298099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6298251Z skip: Skipped due to small world size. (4.202s) 2023-01-11T21:44:22.6298271Z 2023-01-11T21:44:22.6298539Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6298649Z Ran 1 test in 4.202s 2023-01-11T21:44:22.6298733Z 2023-01-11T21:44:22.6298840Z OK (skipped=1) 2023-01-11T21:44:22.6298860Z 2023-01-11T21:44:22.6298981Z Generating XML reports... 2023-01-11T21:44:22.6299424Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212042.xml 2023-01-11T21:44:22.6299773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6299944Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6300315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6300503Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6300522Z 2023-01-11T21:44:22.6300624Z Running tests... 2023-01-11T21:44:22.6300887Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6301190Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6301449Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6301662Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8186 2023-01-11T21:44:22.6301854Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8187 2023-01-11T21:44:22.6302256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6302434Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6302802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6302990Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6303344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6303518Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6303881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6304047Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6304287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6304677Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6304913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6305288Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6305511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6305727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6305882Z skip: Skipped due to small world size. (4.245s) 2023-01-11T21:44:22.6305902Z 2023-01-11T21:44:22.6306162Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6306255Z Ran 1 test in 4.245s 2023-01-11T21:44:22.6306275Z 2023-01-11T21:44:22.6306379Z OK (skipped=1) 2023-01-11T21:44:22.6306398Z 2023-01-11T21:44:22.6306513Z Generating XML reports... 2023-01-11T21:44:22.6306959Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212049.xml 2023-01-11T21:44:22.6307369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6307538Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6307912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6308157Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6308177Z 2023-01-11T21:44:22.6308277Z Running tests... 2023-01-11T21:44:22.6308527Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6308838Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6309106Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6309311Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8295 2023-01-11T21:44:22.6309521Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8296 2023-01-11T21:44:22.6309878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6310050Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6310424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6310593Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6310944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6311114Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6311523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6311706Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6311947Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6312185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6312577Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6312969Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6313176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6313399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6313549Z skip: Skipped due to small world size. (4.224s) 2023-01-11T21:44:22.6313568Z 2023-01-11T21:44:22.6313832Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6313933Z Ran 1 test in 4.224s 2023-01-11T21:44:22.6313953Z 2023-01-11T21:44:22.6314058Z OK (skipped=1) 2023-01-11T21:44:22.6314077Z 2023-01-11T21:44:22.6314190Z Generating XML reports... 2023-01-11T21:44:22.6314637Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212056.xml 2023-01-11T21:44:22.6314992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6315158Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6315526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6315715Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6315734Z 2023-01-11T21:44:22.6315835Z Running tests... 2023-01-11T21:44:22.6316096Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6316404Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6316652Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6316918Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8404 2023-01-11T21:44:22.6317110Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8405 2023-01-11T21:44:22.6317472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6317642Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6318014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6318199Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6318558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6318721Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6319092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6319261Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6319496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6319732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6320164Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6320565Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6320782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6321003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6321159Z skip: Skipped due to small world size. (4.240s) 2023-01-11T21:44:22.6321181Z 2023-01-11T21:44:22.6321437Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6321531Z Ran 1 test in 4.241s 2023-01-11T21:44:22.6321566Z 2023-01-11T21:44:22.6321655Z OK (skipped=1) 2023-01-11T21:44:22.6321674Z 2023-01-11T21:44:22.6321788Z Generating XML reports... 2023-01-11T21:44:22.6322231Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212103.xml 2023-01-11T21:44:22.6322592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6322762Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6323131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6323317Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6323340Z 2023-01-11T21:44:22.6323447Z Running tests... 2023-01-11T21:44:22.6323692Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6323991Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6324238Z test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6324445Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8513 2023-01-11T21:44:22.6324661Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8514 2023-01-11T21:44:22.6325018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6325189Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6325561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6325790Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6326149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6326318Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6326684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6326870Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6327104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6327341Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6327733Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6328122Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6328335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6328553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6328883Z STAGE:2023-01-11 21:21:13 8513:8513 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6329255Z STAGE:2023-01-11 21:21:13 8514:8514 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6329581Z STAGE:2023-01-11 21:21:13 8513:8513 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6329899Z STAGE:2023-01-11 21:21:13 8514:8514 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6330236Z STAGE:2023-01-11 21:21:13 8513:8513 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6330764Z STAGE:2023-01-11 21:21:13 8514:8514 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6331082Z STAGE:2023-01-11 21:21:13 8514:8514 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6331379Z STAGE:2023-01-11 21:21:13 8513:8513 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6331704Z STAGE:2023-01-11 21:21:13 8514:8514 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6332020Z STAGE:2023-01-11 21:21:13 8513:8513 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6332352Z STAGE:2023-01-11 21:21:13 8514:8514 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6332684Z STAGE:2023-01-11 21:21:13 8513:8513 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6332780Z ok (4.231s) 2023-01-11T21:44:22.6332800Z 2023-01-11T21:44:22.6333063Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6333176Z Ran 1 test in 4.231s 2023-01-11T21:44:22.6333196Z 2023-01-11T21:44:22.6333270Z OK 2023-01-11T21:44:22.6333301Z 2023-01-11T21:44:22.6333405Z Generating XML reports... 2023-01-11T21:44:22.6333851Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212109.xml 2023-01-11T21:44:22.6334219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6334392Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6334767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6334953Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6334972Z 2023-01-11T21:44:22.6335078Z Running tests... 2023-01-11T21:44:22.6335341Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6335733Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6335970Z test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6336184Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8626 2023-01-11T21:44:22.6336397Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8627 2023-01-11T21:44:22.6336765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6336937Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6337311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6337497Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6337838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6338012Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6338383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6338569Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6338866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6339114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6339514Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6339902Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6340131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6340337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6340664Z STAGE:2023-01-11 21:21:20 8626:8626 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6340985Z STAGE:2023-01-11 21:21:20 8627:8627 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6341315Z STAGE:2023-01-11 21:21:20 8626:8626 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6341641Z STAGE:2023-01-11 21:21:20 8627:8627 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6342197Z STAGE:2023-01-11 21:21:20 8626:8626 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:21:20 8627:8627 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6342217Z 2023-01-11T21:44:22.6342537Z STAGE:2023-01-11 21:21:20 8627:8627 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6342849Z STAGE:2023-01-11 21:21:20 8626:8626 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6343168Z STAGE:2023-01-11 21:21:20 8626:8626 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6343487Z STAGE:2023-01-11 21:21:20 8627:8627 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6343809Z STAGE:2023-01-11 21:21:20 8626:8626 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6344146Z STAGE:2023-01-11 21:21:20 8627:8627 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6344247Z ok (4.235s) 2023-01-11T21:44:22.6344267Z 2023-01-11T21:44:22.6344532Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6344642Z Ran 1 test in 4.236s 2023-01-11T21:44:22.6344662Z 2023-01-11T21:44:22.6344819Z OK 2023-01-11T21:44:22.6344838Z 2023-01-11T21:44:22.6344959Z Generating XML reports... 2023-01-11T21:44:22.6345410Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212116.xml 2023-01-11T21:44:22.6345757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6345931Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6346307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6346495Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6346514Z 2023-01-11T21:44:22.6346620Z Running tests... 2023-01-11T21:44:22.6346884Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6347190Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6347450Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6347661Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8739 2023-01-11T21:44:22.6347857Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8740 2023-01-11T21:44:22.6348223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6348444Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6348826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6349015Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6349371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6349542Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6349908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6350074Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6350313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6350554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6350945Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6351334Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6351559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6351784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6352107Z STAGE:2023-01-11 21:21:28 8740:8740 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6352869Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1582: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2023-01-11T21:44:22.6352982Z warnings.warn( 2023-01-11T21:44:22.6353287Z STAGE:2023-01-11 21:21:28 8739:8739 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6354034Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1582: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2023-01-11T21:44:22.6354198Z warnings.warn( 2023-01-11T21:44:22.6354532Z STAGE:2023-01-11 21:21:28 8739:8739 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6354850Z STAGE:2023-01-11 21:21:28 8740:8740 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6355189Z STAGE:2023-01-11 21:21:28 8739:8739 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6355528Z STAGE:2023-01-11 21:21:28 8740:8740 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6355846Z STAGE:2023-01-11 21:21:28 8739:8739 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6356161Z STAGE:2023-01-11 21:21:28 8740:8740 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6356468Z STAGE:2023-01-11 21:21:28 8739:8739 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6356790Z STAGE:2023-01-11 21:21:28 8740:8740 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6357128Z STAGE:2023-01-11 21:21:28 8739:8739 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6357460Z STAGE:2023-01-11 21:21:28 8740:8740 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6357560Z ok (5.994s) 2023-01-11T21:44:22.6357579Z 2023-01-11T21:44:22.6357846Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6358003Z Ran 1 test in 5.994s 2023-01-11T21:44:22.6358023Z 2023-01-11T21:44:22.6358114Z OK 2023-01-11T21:44:22.6358133Z 2023-01-11T21:44:22.6358240Z Generating XML reports... 2023-01-11T21:44:22.6358692Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212123.xml 2023-01-11T21:44:22.6359054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6359231Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6359603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6359791Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6359810Z 2023-01-11T21:44:22.6359916Z Running tests... 2023-01-11T21:44:22.6360182Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6360495Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6360747Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6360960Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8854 2023-01-11T21:44:22.6361169Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8855 2023-01-11T21:44:22.6361534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6361709Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6362083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6362271Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6362632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6362784Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6363149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6363333Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6363576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6363873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6364269Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6364659Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6364887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6365110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6365415Z STAGE:2023-01-11 21:21:37 8855:8855 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6366174Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1582: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2023-01-11T21:44:22.6366291Z warnings.warn( 2023-01-11T21:44:22.6366617Z STAGE:2023-01-11 21:21:37 8854:8854 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6367467Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1582: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2023-01-11T21:44:22.6367581Z warnings.warn( 2023-01-11T21:44:22.6367913Z STAGE:2023-01-11 21:21:37 8854:8854 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6368234Z STAGE:2023-01-11 21:21:37 8855:8855 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6368571Z STAGE:2023-01-11 21:21:37 8854:8854 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6368913Z STAGE:2023-01-11 21:21:37 8855:8855 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6369215Z STAGE:2023-01-11 21:21:37 8854:8854 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6369535Z STAGE:2023-01-11 21:21:37 8855:8855 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6369862Z STAGE:2023-01-11 21:21:37 8854:8854 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6370184Z STAGE:2023-01-11 21:21:37 8855:8855 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6370696Z STAGE:2023-01-11 21:21:37 8854:8854 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6371034Z STAGE:2023-01-11 21:21:37 8855:8855 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6371137Z ok (6.046s) 2023-01-11T21:44:22.6371162Z 2023-01-11T21:44:22.6371427Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6371537Z Ran 1 test in 6.047s 2023-01-11T21:44:22.6371557Z 2023-01-11T21:44:22.6371631Z OK 2023-01-11T21:44:22.6371650Z 2023-01-11T21:44:22.6371771Z Generating XML reports... 2023-01-11T21:44:22.6372216Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212131.xml 2023-01-11T21:44:22.6372584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6372757Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6373132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6373318Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6373337Z 2023-01-11T21:44:22.6373565Z Running tests... 2023-01-11T21:44:22.6373817Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6374128Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6374385Z test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6374599Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8969 2023-01-11T21:44:22.6374814Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8970 2023-01-11T21:44:22.6375181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6375355Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6375730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6375917Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6376266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6376437Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6376802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6377049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6377300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6377540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6377937Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6378329Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6378557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6378762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6379087Z STAGE:2023-01-11 21:21:44 8970:8970 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6379410Z STAGE:2023-01-11 21:21:44 8969:8969 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6379738Z STAGE:2023-01-11 21:21:44 8970:8970 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6380057Z STAGE:2023-01-11 21:21:44 8969:8969 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6380391Z STAGE:2023-01-11 21:21:44 8970:8970 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6380720Z STAGE:2023-01-11 21:21:44 8969:8969 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6381038Z STAGE:2023-01-11 21:21:44 8970:8970 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6381337Z STAGE:2023-01-11 21:21:44 8969:8969 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6381660Z STAGE:2023-01-11 21:21:44 8970:8970 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6381981Z STAGE:2023-01-11 21:21:44 8969:8969 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6382318Z STAGE:2023-01-11 21:21:44 8970:8970 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6382654Z STAGE:2023-01-11 21:21:44 8969:8969 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6382755Z ok (4.232s) 2023-01-11T21:44:22.6382775Z 2023-01-11T21:44:22.6383042Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6383208Z Ran 1 test in 4.232s 2023-01-11T21:44:22.6383228Z 2023-01-11T21:44:22.6383320Z OK 2023-01-11T21:44:22.6383339Z 2023-01-11T21:44:22.6383444Z Generating XML reports... 2023-01-11T21:44:22.6383893Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212140.xml 2023-01-11T21:44:22.6384259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6384433Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6384804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6384989Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6385009Z 2023-01-11T21:44:22.6385116Z Running tests... 2023-01-11T21:44:22.6385380Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6385678Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6385939Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6386154Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9082 2023-01-11T21:44:22.6386366Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9083 2023-01-11T21:44:22.6386779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6386956Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6387333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6387519Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6387874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6388031Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6388396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6388577Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6388820Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6389059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6389454Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6389843Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6390065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6390292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6390378Z ok (5.778s) 2023-01-11T21:44:22.6390397Z 2023-01-11T21:44:22.6390661Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6390771Z Ran 1 test in 5.778s 2023-01-11T21:44:22.6390790Z 2023-01-11T21:44:22.6390881Z OK 2023-01-11T21:44:22.6390900Z 2023-01-11T21:44:22.6391025Z Generating XML reports... 2023-01-11T21:44:22.6391471Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212147.xml 2023-01-11T21:44:22.6391839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6392012Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6392368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6392611Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6392631Z 2023-01-11T21:44:22.6392737Z Running tests... 2023-01-11T21:44:22.6393001Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6393310Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6393561Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6393774Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9193 2023-01-11T21:44:22.6393983Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9194 2023-01-11T21:44:22.6394334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6394505Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6394883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6395067Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6395425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6395593Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6396004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6396192Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6396433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6396812Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6397049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6397442Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6397665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6397891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6398215Z STAGE:2023-01-11 21:21:59 9194:9194 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6398538Z STAGE:2023-01-11 21:21:59 9193:9193 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6398860Z STAGE:2023-01-11 21:21:59 9194:9194 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6399178Z STAGE:2023-01-11 21:21:59 9193:9193 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6399502Z STAGE:2023-01-11 21:21:59 9194:9194 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6399832Z STAGE:2023-01-11 21:21:59 9193:9193 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6400148Z STAGE:2023-01-11 21:21:59 9194:9194 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6400468Z STAGE:2023-01-11 21:21:59 9193:9193 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6400791Z STAGE:2023-01-11 21:21:59 9194:9194 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6401109Z STAGE:2023-01-11 21:21:59 9193:9193 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6401445Z STAGE:2023-01-11 21:21:59 9194:9194 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6401781Z STAGE:2023-01-11 21:21:59 9193:9193 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6401940Z ok (4.195s) 2023-01-11T21:44:22.6401960Z 2023-01-11T21:44:22.6402209Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6402319Z Ran 1 test in 4.195s 2023-01-11T21:44:22.6402340Z 2023-01-11T21:44:22.6402430Z OK 2023-01-11T21:44:22.6402449Z 2023-01-11T21:44:22.6402571Z Generating XML reports... 2023-01-11T21:44:22.6403019Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212155.xml 2023-01-11T21:44:22.6403383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6403555Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6403928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6404097Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6404136Z 2023-01-11T21:44:22.6404225Z Running tests... 2023-01-11T21:44:22.6404486Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6404794Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6405052Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6405316Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9306 2023-01-11T21:44:22.6405532Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9307 2023-01-11T21:44:22.6405900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6406071Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6406428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6406619Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6406976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6407145Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6407515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6407698Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6407936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6408330Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6408550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6408940Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6409164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6409384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6409715Z STAGE:2023-01-11 21:22:06 9307:9307 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6410038Z STAGE:2023-01-11 21:22:06 9306:9306 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6410747Z STAGE:2023-01-11 21:22:06 9307:9307 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:22:06 9306:9306 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6410770Z 2023-01-11T21:44:22.6411334Z STAGE:2023-01-11 21:22:06 9306:9306 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:22:06 9307:9307 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6411441Z 2023-01-11T21:44:22.6411967Z STAGE:2023-01-11 21:22:06 9306:9306 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:22:06 9307:9307 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6411987Z 2023-01-11T21:44:22.6412314Z STAGE:2023-01-11 21:22:06 9307:9307 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6412632Z STAGE:2023-01-11 21:22:06 9306:9306 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6412965Z STAGE:2023-01-11 21:22:06 9307:9307 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6413304Z STAGE:2023-01-11 21:22:06 9306:9306 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6413397Z ok (4.268s) 2023-01-11T21:44:22.6413408Z 2023-01-11T21:44:22.6413672Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6413784Z Ran 1 test in 4.269s 2023-01-11T21:44:22.6413803Z 2023-01-11T21:44:22.6413894Z OK 2023-01-11T21:44:22.6413913Z 2023-01-11T21:44:22.6414035Z Generating XML reports... 2023-01-11T21:44:22.6414485Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212202.xml 2023-01-11T21:44:22.6414913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6415079Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6415458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6415645Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6415664Z 2023-01-11T21:44:22.6415772Z Running tests... 2023-01-11T21:44:22.6416039Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6416346Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6416609Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6416822Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9419 2023-01-11T21:44:22.6417019Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9420 2023-01-11T21:44:22.6417387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6417558Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6417931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6418115Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6418475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6418643Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6419010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6419196Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6419419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6419815Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6420046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6420437Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6420716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6420939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6421266Z STAGE:2023-01-11 21:22:13 9419:9419 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6421591Z STAGE:2023-01-11 21:22:13 9420:9420 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6422124Z STAGE:2023-01-11 21:22:13 9420:9420 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:22:13 9419:9419 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6422144Z 2023-01-11T21:44:22.6422699Z STAGE:2023-01-11 21:22:13 9419:9419 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:22:13 9420:9420 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6422723Z 2023-01-11T21:44:22.6423040Z STAGE:2023-01-11 21:22:13 9420:9420 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6423340Z STAGE:2023-01-11 21:22:13 9419:9419 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6423661Z STAGE:2023-01-11 21:22:13 9420:9420 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6424023Z STAGE:2023-01-11 21:22:13 9419:9419 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6424368Z STAGE:2023-01-11 21:22:13 9420:9420 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6424696Z STAGE:2023-01-11 21:22:13 9419:9419 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6424797Z ok (4.200s) 2023-01-11T21:44:22.6424817Z 2023-01-11T21:44:22.6425083Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6425197Z Ran 1 test in 4.200s 2023-01-11T21:44:22.6425217Z 2023-01-11T21:44:22.6425290Z OK 2023-01-11T21:44:22.6425309Z 2023-01-11T21:44:22.6425430Z Generating XML reports... 2023-01-11T21:44:22.6425877Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212209.xml 2023-01-11T21:44:22.6426245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6426420Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6426795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6426983Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6427003Z 2023-01-11T21:44:22.6427108Z Running tests... 2023-01-11T21:44:22.6427373Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6427671Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6427925Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6428139Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9532 2023-01-11T21:44:22.6428350Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9533 2023-01-11T21:44:22.6428715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6428888Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6429262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6429448Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6429789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6430014Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6430389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6430574Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6430817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6431056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6431451Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6431843Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6432069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6432279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6432604Z STAGE:2023-01-11 21:22:21 9533:9533 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6432927Z STAGE:2023-01-11 21:22:21 9532:9532 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6433299Z STAGE:2023-01-11 21:22:21 9532:9532 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6433631Z STAGE:2023-01-11 21:22:21 9533:9533 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6433970Z STAGE:2023-01-11 21:22:21 9532:9532 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6434308Z STAGE:2023-01-11 21:22:21 9533:9533 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6434823Z STAGE:2023-01-11 21:22:21 9533:9533 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:22:21 9532:9532 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6434846Z 2023-01-11T21:44:22.6435373Z STAGE:2023-01-11 21:22:23 9532:9532 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:22:23 9533:9533 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6435392Z 2023-01-11T21:44:22.6435950Z STAGE:2023-01-11 21:22:23 9533:9533 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:22:23 9532:9532 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6435970Z 2023-01-11T21:44:22.6436291Z STAGE:2023-01-11 21:22:23 9533:9533 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6436588Z STAGE:2023-01-11 21:22:23 9532:9532 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6437112Z STAGE:2023-01-11 21:22:23 9532:9532 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:22:23 9533:9533 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6437135Z 2023-01-11T21:44:22.6437473Z STAGE:2023-01-11 21:22:23 9533:9533 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6437806Z STAGE:2023-01-11 21:22:23 9532:9532 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6437906Z ok (7.564s) 2023-01-11T21:44:22.6437929Z 2023-01-11T21:44:22.6438196Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6438306Z Ran 1 test in 7.565s 2023-01-11T21:44:22.6438325Z 2023-01-11T21:44:22.6438416Z OK 2023-01-11T21:44:22.6438435Z 2023-01-11T21:44:22.6438555Z Generating XML reports... 2023-01-11T21:44:22.6438983Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212216.xml 2023-01-11T21:44:22.6439408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6439580Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6439956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6440141Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6440161Z 2023-01-11T21:44:22.6440270Z Running tests... 2023-01-11T21:44:22.6440532Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6440841Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6441103Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6441301Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9648 2023-01-11T21:44:22.6441516Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9649 2023-01-11T21:44:22.6441882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6442052Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6442427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6442659Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6443030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6443198Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6443550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6443734Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6443976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6444215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6444610Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6445001Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6445226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6445447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6445771Z STAGE:2023-01-11 21:22:31 9649:9649 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6446072Z STAGE:2023-01-11 21:22:31 9648:9648 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6446403Z STAGE:2023-01-11 21:22:31 9648:9648 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6446727Z STAGE:2023-01-11 21:22:31 9649:9649 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6447061Z STAGE:2023-01-11 21:22:31 9648:9648 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6447398Z STAGE:2023-01-11 21:22:31 9649:9649 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6447906Z STAGE:2023-01-11 21:22:31 9649:9649 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:22:31 9648:9648 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6447926Z 2023-01-11T21:44:22.6448450Z STAGE:2023-01-11 21:22:32 9649:9649 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:22:32 9648:9648 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6448518Z 2023-01-11T21:44:22.6449075Z STAGE:2023-01-11 21:22:32 9649:9649 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:22:32 9648:9648 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6449095Z 2023-01-11T21:44:22.6449607Z STAGE:2023-01-11 21:22:32 9649:9649 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:22:32 9648:9648 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6449627Z 2023-01-11T21:44:22.6449949Z STAGE:2023-01-11 21:22:32 9648:9648 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6450267Z STAGE:2023-01-11 21:22:32 9649:9649 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6450993Z STAGE:2023-01-11 21:22:32 9648:9648 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:22:32 9649:9649 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6451020Z 2023-01-11T21:44:22.6451122Z ok (7.427s) 2023-01-11T21:44:22.6451141Z 2023-01-11T21:44:22.6451391Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6451501Z Ran 1 test in 7.427s 2023-01-11T21:44:22.6451521Z 2023-01-11T21:44:22.6451610Z OK 2023-01-11T21:44:22.6451628Z 2023-01-11T21:44:22.6451751Z Generating XML reports... 2023-01-11T21:44:22.6452266Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212226.xml 2023-01-11T21:44:22.6452647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6452821Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6453198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6453390Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6453410Z 2023-01-11T21:44:22.6453501Z Running tests... 2023-01-11T21:44:22.6453768Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6454076Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6454348Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6454561Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9764 2023-01-11T21:44:22.6454771Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9765 2023-01-11T21:44:22.6455135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6455307Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6455663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6455853Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6456210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6456377Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6456745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6456929Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6457169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6457409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6457804Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6458255Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6458481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6458704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6459032Z STAGE:2023-01-11 21:22:41 9765:9765 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6459355Z STAGE:2023-01-11 21:22:41 9764:9764 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6459680Z STAGE:2023-01-11 21:22:41 9764:9764 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6460000Z STAGE:2023-01-11 21:22:41 9765:9765 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6460341Z STAGE:2023-01-11 21:22:41 9764:9764 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6460653Z STAGE:2023-01-11 21:22:41 9765:9765 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6460968Z STAGE:2023-01-11 21:22:41 9765:9765 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6461284Z STAGE:2023-01-11 21:22:41 9764:9764 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6461858Z STAGE:2023-01-11 21:22:43 9765:9765 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:22:43 9764:9764 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6461881Z 2023-01-11T21:44:22.6462442Z STAGE:2023-01-11 21:22:43 9765:9765 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:22:43 9764:9764 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6462462Z 2023-01-11T21:44:22.6462979Z STAGE:2023-01-11 21:22:43 9765:9765 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:22:43 9764:9764 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6462999Z 2023-01-11T21:44:22.6463319Z STAGE:2023-01-11 21:22:43 9764:9764 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6463639Z STAGE:2023-01-11 21:22:43 9765:9765 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6463974Z STAGE:2023-01-11 21:22:43 9764:9764 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6464311Z STAGE:2023-01-11 21:22:43 9765:9765 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6464412Z ok (7.443s) 2023-01-11T21:44:22.6464431Z 2023-01-11T21:44:22.6464697Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6464791Z Ran 1 test in 7.444s 2023-01-11T21:44:22.6464811Z 2023-01-11T21:44:22.6464905Z OK 2023-01-11T21:44:22.6464924Z 2023-01-11T21:44:22.6465046Z Generating XML reports... 2023-01-11T21:44:22.6465493Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212236.xml 2023-01-11T21:44:22.6465862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6466034Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6466413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6466600Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6466620Z 2023-01-11T21:44:22.6466710Z Running tests... 2023-01-11T21:44:22.6466976Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6467285Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6467581Z test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s) 2023-01-11T21:44:22.6467601Z 2023-01-11T21:44:22.6467863Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6467974Z Ran 1 test in 0.002s 2023-01-11T21:44:22.6467993Z 2023-01-11T21:44:22.6468102Z OK (skipped=1) 2023-01-11T21:44:22.6468121Z 2023-01-11T21:44:22.6468243Z Generating XML reports... 2023-01-11T21:44:22.6468685Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212246.xml 2023-01-11T21:44:22.6469035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6469206Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6469577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6469768Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6469787Z 2023-01-11T21:44:22.6469894Z Running tests... 2023-01-11T21:44:22.6470156Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6470464Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6470762Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s) 2023-01-11T21:44:22.6470783Z 2023-01-11T21:44:22.6471047Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6471141Z Ran 1 test in 0.002s 2023-01-11T21:44:22.6471160Z 2023-01-11T21:44:22.6471266Z OK (skipped=1) 2023-01-11T21:44:22.6471286Z 2023-01-11T21:44:22.6471407Z Generating XML reports... 2023-01-11T21:44:22.6471848Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212248.xml 2023-01-11T21:44:22.6472217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6472389Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6472763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6472951Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6472974Z 2023-01-11T21:44:22.6473064Z Running tests... 2023-01-11T21:44:22.6473326Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6473679Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6473932Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s) 2023-01-11T21:44:22.6473952Z 2023-01-11T21:44:22.6474216Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6474330Z Ran 1 test in 0.002s 2023-01-11T21:44:22.6474349Z 2023-01-11T21:44:22.6474456Z OK (skipped=1) 2023-01-11T21:44:22.6474475Z 2023-01-11T21:44:22.6474597Z Generating XML reports... 2023-01-11T21:44:22.6475041Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212250.xml 2023-01-11T21:44:22.6475392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6475567Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6475943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6476129Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6476149Z 2023-01-11T21:44:22.6476254Z Running tests... 2023-01-11T21:44:22.6476517Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6476897Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6477160Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s) 2023-01-11T21:44:22.6477179Z 2023-01-11T21:44:22.6477438Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6477531Z Ran 1 test in 0.002s 2023-01-11T21:44:22.6477552Z 2023-01-11T21:44:22.6477660Z OK (skipped=1) 2023-01-11T21:44:22.6477678Z 2023-01-11T21:44:22.6477799Z Generating XML reports... 2023-01-11T21:44:22.6478240Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212253.xml 2023-01-11T21:44:22.6478604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6478776Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6479152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6479338Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6479358Z 2023-01-11T21:44:22.6479447Z Running tests... 2023-01-11T21:44:22.6479709Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6480065Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6480326Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s) 2023-01-11T21:44:22.6480345Z 2023-01-11T21:44:22.6480606Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6480714Z Ran 1 test in 0.002s 2023-01-11T21:44:22.6480733Z 2023-01-11T21:44:22.6480838Z OK (skipped=1) 2023-01-11T21:44:22.6480856Z 2023-01-11T21:44:22.6480981Z Generating XML reports... 2023-01-11T21:44:22.6481422Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212255.xml 2023-01-11T21:44:22.6481766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6481939Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6482315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6482501Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6482520Z 2023-01-11T21:44:22.6482626Z Running tests... 2023-01-11T21:44:22.6482889Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6483195Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6483463Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s) 2023-01-11T21:44:22.6483486Z 2023-01-11T21:44:22.6483744Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6483838Z Ran 1 test in 0.002s 2023-01-11T21:44:22.6483857Z 2023-01-11T21:44:22.6483963Z OK (skipped=1) 2023-01-11T21:44:22.6483982Z 2023-01-11T21:44:22.6484103Z Generating XML reports... 2023-01-11T21:44:22.6484548Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212258.xml 2023-01-11T21:44:22.6484913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6485088Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6485459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6485701Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6485720Z 2023-01-11T21:44:22.6485826Z Running tests... 2023-01-11T21:44:22.6486072Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6486383Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6486631Z test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s) 2023-01-11T21:44:22.6486654Z 2023-01-11T21:44:22.6486913Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6487022Z Ran 1 test in 0.002s 2023-01-11T21:44:22.6487041Z 2023-01-11T21:44:22.6487145Z OK (skipped=1) 2023-01-11T21:44:22.6487164Z 2023-01-11T21:44:22.6487283Z Generating XML reports... 2023-01-11T21:44:22.6487722Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212300.xml 2023-01-11T21:44:22.6488072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6488244Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6488617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6488802Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6488822Z 2023-01-11T21:44:22.6488974Z Running tests... 2023-01-11T21:44:22.6489238Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6489545Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6489814Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2023-01-11T21:44:22.6489833Z 2023-01-11T21:44:22.6490091Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6490188Z Ran 1 test in 0.002s 2023-01-11T21:44:22.6490207Z 2023-01-11T21:44:22.6490484Z OK (skipped=1) 2023-01-11T21:44:22.6490492Z 2023-01-11T21:44:22.6490621Z Generating XML reports... 2023-01-11T21:44:22.6491067Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212302.xml 2023-01-11T21:44:22.6491433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6491604Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6491979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6492165Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6492184Z 2023-01-11T21:44:22.6492290Z Running tests... 2023-01-11T21:44:22.6492536Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6492849Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6493128Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s) 2023-01-11T21:44:22.6493149Z 2023-01-11T21:44:22.6493415Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6493525Z Ran 1 test in 0.002s 2023-01-11T21:44:22.6493544Z 2023-01-11T21:44:22.6493652Z OK (skipped=1) 2023-01-11T21:44:22.6493671Z 2023-01-11T21:44:22.6493792Z Generating XML reports... 2023-01-11T21:44:22.6494231Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212305.xml 2023-01-11T21:44:22.6494593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6494747Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6495208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6495395Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6495414Z 2023-01-11T21:44:22.6495520Z Running tests... 2023-01-11T21:44:22.6495778Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6496089Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6496377Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s) 2023-01-11T21:44:22.6496397Z 2023-01-11T21:44:22.6496655Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6496748Z Ran 1 test in 0.002s 2023-01-11T21:44:22.6496785Z 2023-01-11T21:44:22.6496873Z OK (skipped=1) 2023-01-11T21:44:22.6496896Z 2023-01-11T21:44:22.6497017Z Generating XML reports... 2023-01-11T21:44:22.6497455Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212307.xml 2023-01-11T21:44:22.6497817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6497991Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6498424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6498619Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6498638Z 2023-01-11T21:44:22.6498743Z Running tests... 2023-01-11T21:44:22.6498990Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6499297Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6499589Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2023-01-11T21:44:22.6499608Z 2023-01-11T21:44:22.6499865Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6499975Z Ran 1 test in 0.002s 2023-01-11T21:44:22.6499993Z 2023-01-11T21:44:22.6500098Z OK (skipped=1) 2023-01-11T21:44:22.6500117Z 2023-01-11T21:44:22.6500237Z Generating XML reports... 2023-01-11T21:44:22.6500679Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212310.xml 2023-01-11T21:44:22.6501040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6501195Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6501566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6501757Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6501775Z 2023-01-11T21:44:22.6501881Z Running tests... 2023-01-11T21:44:22.6502141Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6502446Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6502743Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2023-01-11T21:44:22.6502764Z 2023-01-11T21:44:22.6503021Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6503130Z Ran 1 test in 0.002s 2023-01-11T21:44:22.6503150Z 2023-01-11T21:44:22.6503239Z OK (skipped=1) 2023-01-11T21:44:22.6503258Z 2023-01-11T21:44:22.6503379Z Generating XML reports... 2023-01-11T21:44:22.6503818Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212312.xml 2023-01-11T21:44:22.6504237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6504408Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6504778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6504965Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6504984Z 2023-01-11T21:44:22.6505090Z Running tests... 2023-01-11T21:44:22.6505333Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6505640Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6505927Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s) 2023-01-11T21:44:22.6505950Z 2023-01-11T21:44:22.6506209Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6506319Z Ran 1 test in 0.002s 2023-01-11T21:44:22.6506338Z 2023-01-11T21:44:22.6506443Z OK (skipped=1) 2023-01-11T21:44:22.6506462Z 2023-01-11T21:44:22.6506583Z Generating XML reports... 2023-01-11T21:44:22.6507023Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212314.xml 2023-01-11T21:44:22.6507437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6507597Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6507973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6508158Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6508180Z 2023-01-11T21:44:22.6508290Z Running tests... 2023-01-11T21:44:22.6508552Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6508860Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6509159Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2023-01-11T21:44:22.6509178Z 2023-01-11T21:44:22.6509440Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6509550Z Ran 1 test in 0.002s 2023-01-11T21:44:22.6509569Z 2023-01-11T21:44:22.6509658Z OK (skipped=1) 2023-01-11T21:44:22.6509693Z 2023-01-11T21:44:22.6509797Z Generating XML reports... 2023-01-11T21:44:22.6510234Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212317.xml 2023-01-11T21:44:22.6510599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6510774Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6511146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6511331Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6511351Z 2023-01-11T21:44:22.6511457Z Running tests... 2023-01-11T21:44:22.6511723Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6512015Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6512301Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s) 2023-01-11T21:44:22.6512320Z 2023-01-11T21:44:22.6512580Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6512690Z Ran 1 test in 0.002s 2023-01-11T21:44:22.6512814Z 2023-01-11T21:44:22.6512926Z OK (skipped=1) 2023-01-11T21:44:22.6512945Z 2023-01-11T21:44:22.6513066Z Generating XML reports... 2023-01-11T21:44:22.6513512Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212319.xml 2023-01-11T21:44:22.6513876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6514051Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6514407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6514594Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6514613Z 2023-01-11T21:44:22.6514719Z Running tests... 2023-01-11T21:44:22.6514979Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6515286Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6515582Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2023-01-11T21:44:22.6515602Z 2023-01-11T21:44:22.6515860Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6515969Z Ran 1 test in 0.002s 2023-01-11T21:44:22.6515988Z 2023-01-11T21:44:22.6516094Z OK (skipped=1) 2023-01-11T21:44:22.6516113Z 2023-01-11T21:44:22.6516267Z Generating XML reports... 2023-01-11T21:44:22.6516715Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212322.xml 2023-01-11T21:44:22.6517079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6517252Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6517624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6517815Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6517834Z 2023-01-11T21:44:22.6517940Z Running tests... 2023-01-11T21:44:22.6518200Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6518489Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6518772Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s) 2023-01-11T21:44:22.6518793Z 2023-01-11T21:44:22.6519051Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6519159Z Ran 1 test in 0.002s 2023-01-11T21:44:22.6519178Z 2023-01-11T21:44:22.6519281Z OK (skipped=1) 2023-01-11T21:44:22.6519300Z 2023-01-11T21:44:22.6519421Z Generating XML reports... 2023-01-11T21:44:22.6519863Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212324.xml 2023-01-11T21:44:22.6520228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6520398Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6520747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6520936Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6520955Z 2023-01-11T21:44:22.6521061Z Running tests... 2023-01-11T21:44:22.6521321Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6521630Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6521923Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s) 2023-01-11T21:44:22.6521992Z 2023-01-11T21:44:22.6522255Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6522365Z Ran 1 test in 0.002s 2023-01-11T21:44:22.6522384Z 2023-01-11T21:44:22.6522489Z OK (skipped=1) 2023-01-11T21:44:22.6522508Z 2023-01-11T21:44:22.6522612Z Generating XML reports... 2023-01-11T21:44:22.6523061Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212326.xml 2023-01-11T21:44:22.6523425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6523596Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6523969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6524157Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6524179Z 2023-01-11T21:44:22.6524287Z Running tests... 2023-01-11T21:44:22.6524547Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6524836Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6525124Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2023-01-11T21:44:22.6525144Z 2023-01-11T21:44:22.6525447Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6525562Z Ran 1 test in 0.002s 2023-01-11T21:44:22.6525581Z 2023-01-11T21:44:22.6525685Z OK (skipped=1) 2023-01-11T21:44:22.6525704Z 2023-01-11T21:44:22.6525825Z Generating XML reports... 2023-01-11T21:44:22.6526263Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212329.xml 2023-01-11T21:44:22.6526627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6526803Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6527156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6527341Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6527360Z 2023-01-11T21:44:22.6527467Z Running tests... 2023-01-11T21:44:22.6527733Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6528043Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6528340Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2023-01-11T21:44:22.6528360Z 2023-01-11T21:44:22.6528619Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6528731Z Ran 1 test in 0.002s 2023-01-11T21:44:22.6528749Z 2023-01-11T21:44:22.6528855Z OK (skipped=1) 2023-01-11T21:44:22.6528875Z 2023-01-11T21:44:22.6528979Z Generating XML reports... 2023-01-11T21:44:22.6529417Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212331.xml 2023-01-11T21:44:22.6529783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6529953Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6530484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6530683Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6530703Z 2023-01-11T21:44:22.6530811Z Running tests... 2023-01-11T21:44:22.6531076Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6531471Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6531749Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s) 2023-01-11T21:44:22.6531785Z 2023-01-11T21:44:22.6532030Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6532139Z Ran 1 test in 0.002s 2023-01-11T21:44:22.6532162Z 2023-01-11T21:44:22.6532267Z OK (skipped=1) 2023-01-11T21:44:22.6532286Z 2023-01-11T21:44:22.6532405Z Generating XML reports... 2023-01-11T21:44:22.6532846Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212334.xml 2023-01-11T21:44:22.6533210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6533380Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6533756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6533926Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6533946Z 2023-01-11T21:44:22.6534053Z Running tests... 2023-01-11T21:44:22.6534317Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6534688Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6535001Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2023-01-11T21:44:22.6535021Z 2023-01-11T21:44:22.6535283Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6535393Z Ran 1 test in 0.002s 2023-01-11T21:44:22.6535412Z 2023-01-11T21:44:22.6535516Z OK (skipped=1) 2023-01-11T21:44:22.6535539Z 2023-01-11T21:44:22.6535660Z Generating XML reports... 2023-01-11T21:44:22.6536080Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212336.xml 2023-01-11T21:44:22.6536443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6536615Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6536989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6537176Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6537195Z 2023-01-11T21:44:22.6537299Z Running tests... 2023-01-11T21:44:22.6537561Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6537867Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6538140Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s) 2023-01-11T21:44:22.6538177Z 2023-01-11T21:44:22.6538418Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6538529Z Ran 1 test in 0.002s 2023-01-11T21:44:22.6538547Z 2023-01-11T21:44:22.6538652Z OK (skipped=1) 2023-01-11T21:44:22.6538671Z 2023-01-11T21:44:22.6538791Z Generating XML reports... 2023-01-11T21:44:22.6539235Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212338.xml 2023-01-11T21:44:22.6539601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6539772Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6540144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6540383Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6540420Z 2023-01-11T21:44:22.6540510Z Running tests... 2023-01-11T21:44:22.6540772Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6541081Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6541380Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2023-01-11T21:44:22.6541400Z 2023-01-11T21:44:22.6541658Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6541765Z Ran 1 test in 0.002s 2023-01-11T21:44:22.6541783Z 2023-01-11T21:44:22.6541889Z OK (skipped=1) 2023-01-11T21:44:22.6541908Z 2023-01-11T21:44:22.6542028Z Generating XML reports... 2023-01-11T21:44:22.6542447Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212341.xml 2023-01-11T21:44:22.6542818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6542989Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6543361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6543591Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6543612Z 2023-01-11T21:44:22.6543718Z Running tests... 2023-01-11T21:44:22.6543975Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6544282Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6544543Z test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6544742Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10672 2023-01-11T21:44:22.6544961Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10673 2023-01-11T21:44:22.6545328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6545499Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6545871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6546056Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6546417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6546585Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6546933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6547120Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6547360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6547598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6548000Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6548393Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6548618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6548842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6549075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T21:44:22.6549355Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T21:44:22.6549749Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.6550138Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.6550238Z ok (6.971s) 2023-01-11T21:44:22.6550261Z 2023-01-11T21:44:22.6550528Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6550638Z Ran 1 test in 6.971s 2023-01-11T21:44:22.6550658Z 2023-01-11T21:44:22.6550748Z OK 2023-01-11T21:44:22.6550767Z 2023-01-11T21:44:22.6550889Z Generating XML reports... 2023-01-11T21:44:22.6551333Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212343.xml 2023-01-11T21:44:22.6551687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6551860Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6552237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6552424Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6552443Z 2023-01-11T21:44:22.6552596Z Running tests... 2023-01-11T21:44:22.6552866Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6553176Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6553435Z test_backend_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6553634Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10793 2023-01-11T21:44:22.6553854Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10794 2023-01-11T21:44:22.6554221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6554392Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6554766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6554956Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6555315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6555482Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6555848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6556015Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6556262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6556501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6556896Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6557290Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6557516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6557739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6557886Z skip: Need at least 3 CUDA devices (4.202s) 2023-01-11T21:44:22.6557906Z 2023-01-11T21:44:22.6558167Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6558314Z Ran 1 test in 4.202s 2023-01-11T21:44:22.6558333Z 2023-01-11T21:44:22.6558439Z OK (skipped=1) 2023-01-11T21:44:22.6558458Z 2023-01-11T21:44:22.6558580Z Generating XML reports... 2023-01-11T21:44:22.6559027Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212353.xml 2023-01-11T21:44:22.6559396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6559565Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6559937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6560121Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6560140Z 2023-01-11T21:44:22.6560229Z Running tests... 2023-01-11T21:44:22.6560490Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6560802Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6561051Z test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 3 (0.002s) 2023-01-11T21:44:22.6561071Z 2023-01-11T21:44:22.6561331Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6561441Z Ran 1 test in 0.002s 2023-01-11T21:44:22.6561461Z 2023-01-11T21:44:22.6561612Z OK (skipped=1) 2023-01-11T21:44:22.6561632Z 2023-01-11T21:44:22.6561758Z Generating XML reports... 2023-01-11T21:44:22.6562205Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212400.xml 2023-01-11T21:44:22.6562551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6562723Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6563102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6563288Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6563307Z 2023-01-11T21:44:22.6563413Z Running tests... 2023-01-11T21:44:22.6563671Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6563984Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6564222Z test_barrier (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6564419Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10935 2023-01-11T21:44:22.6564633Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10936 2023-01-11T21:44:22.6564998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6565173Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6565544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6565729Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6566088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6566263Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6566630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6566796Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6567034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6567273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6567725Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6568117Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6568342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6568568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6568669Z ok (5.051s) 2023-01-11T21:44:22.6568689Z 2023-01-11T21:44:22.6568954Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6569048Z Ran 1 test in 5.052s 2023-01-11T21:44:22.6569067Z 2023-01-11T21:44:22.6569158Z OK 2023-01-11T21:44:22.6569177Z 2023-01-11T21:44:22.6569299Z Generating XML reports... 2023-01-11T21:44:22.6569744Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212402.xml 2023-01-11T21:44:22.6570111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6570450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6570841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6571100Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6571123Z 2023-01-11T21:44:22.6571217Z Running tests... 2023-01-11T21:44:22.6571484Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6571794Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6572040Z test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6572258Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11044 2023-01-11T21:44:22.6572471Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11045 2023-01-11T21:44:22.6572839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6573010Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6573387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6573594Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6573957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6574126Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6574493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6574681Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6574921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6575161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6575558Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6575929Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6576154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6576377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6576477Z ok (6.521s) 2023-01-11T21:44:22.6576564Z 2023-01-11T21:44:22.6576836Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6576946Z Ran 1 test in 6.521s 2023-01-11T21:44:22.6576965Z 2023-01-11T21:44:22.6577055Z OK 2023-01-11T21:44:22.6577074Z 2023-01-11T21:44:22.6577196Z Generating XML reports... 2023-01-11T21:44:22.6577643Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212410.xml 2023-01-11T21:44:22.6577993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6578165Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6578536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6578722Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6578742Z 2023-01-11T21:44:22.6578852Z Running tests... 2023-01-11T21:44:22.6579114Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6579422Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6579677Z test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6579876Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11155 2023-01-11T21:44:22.6580139Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11156 2023-01-11T21:44:22.6580516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6580688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6581056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6581242Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6581602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6581770Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6582136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6582308Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6582546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6582787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6583179Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6583565Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6583794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6584016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6584251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T21:44:22.6584490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T21:44:22.6584867Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.6585251Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.6585352Z ok (5.007s) 2023-01-11T21:44:22.6585372Z 2023-01-11T21:44:22.6585636Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6585804Z Ran 1 test in 5.008s 2023-01-11T21:44:22.6585823Z 2023-01-11T21:44:22.6585915Z OK 2023-01-11T21:44:22.6585934Z 2023-01-11T21:44:22.6586056Z Generating XML reports... 2023-01-11T21:44:22.6586506Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212419.xml 2023-01-11T21:44:22.6586858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6587033Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6587410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6587599Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6587618Z 2023-01-11T21:44:22.6587724Z Running tests... 2023-01-11T21:44:22.6587984Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6588299Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6588561Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6588776Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11270 2023-01-11T21:44:22.6588970Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11271 2023-01-11T21:44:22.6589380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6589558Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6589933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6590119Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6590476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6590650Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6591021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6591188Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6591428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6591664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6592056Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6592447Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6592674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6592895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6593052Z skip: Skipped due to small world size. (4.218s) 2023-01-11T21:44:22.6593072Z 2023-01-11T21:44:22.6593339Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6593433Z Ran 1 test in 4.218s 2023-01-11T21:44:22.6593469Z 2023-01-11T21:44:22.6593563Z OK (skipped=1) 2023-01-11T21:44:22.6593582Z 2023-01-11T21:44:22.6593702Z Generating XML reports... 2023-01-11T21:44:22.6594148Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212426.xml 2023-01-11T21:44:22.6594518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6594689Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6595118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6595306Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6595326Z 2023-01-11T21:44:22.6595434Z Running tests... 2023-01-11T21:44:22.6595681Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6595993Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6596242Z test_barrier_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6596457Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11379 2023-01-11T21:44:22.6596671Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11380 2023-01-11T21:44:22.6597035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6597210Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6597586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6597756Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6598115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6598343Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6598723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6598906Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6599147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6599392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6599785Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6600175Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6600383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6600607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6600764Z skip: Skipped due to small world size. (4.198s) 2023-01-11T21:44:22.6600783Z 2023-01-11T21:44:22.6601048Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6601160Z Ran 1 test in 4.198s 2023-01-11T21:44:22.6601179Z 2023-01-11T21:44:22.6601284Z OK (skipped=1) 2023-01-11T21:44:22.6601303Z 2023-01-11T21:44:22.6601428Z Generating XML reports... 2023-01-11T21:44:22.6601872Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212433.xml 2023-01-11T21:44:22.6602238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6602392Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6602768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6602954Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6602973Z 2023-01-11T21:44:22.6603080Z Running tests... 2023-01-11T21:44:22.6603345Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6603652Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6603905Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6604189Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11488 2023-01-11T21:44:22.6604383Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11489 2023-01-11T21:44:22.6604751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6604926Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6605299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6605484Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6605842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6606012Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6606385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6606571Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6606792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6607030Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6607524Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6607930Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6608154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6608375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6608534Z skip: Skipped due to small world size. (4.210s) 2023-01-11T21:44:22.6608555Z 2023-01-11T21:44:22.6608820Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6608914Z Ran 1 test in 4.211s 2023-01-11T21:44:22.6608949Z 2023-01-11T21:44:22.6609038Z OK (skipped=1) 2023-01-11T21:44:22.6609057Z 2023-01-11T21:44:22.6609179Z Generating XML reports... 2023-01-11T21:44:22.6609628Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212440.xml 2023-01-11T21:44:22.6609995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6610166Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6610763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6610956Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6610976Z 2023-01-11T21:44:22.6611083Z Running tests... 2023-01-11T21:44:22.6611332Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6611640Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6611903Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6612123Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11597 2023-01-11T21:44:22.6612335Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11598 2023-01-11T21:44:22.6612700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6612871Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6613242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6613513Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6613858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6614027Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6614399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6614582Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6614822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6615060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6615453Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6615850Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6616059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6616283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6616574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T21:44:22.6616818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T21:44:22.6617214Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.6617601Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.6617705Z ok (5.343s) 2023-01-11T21:44:22.6617725Z 2023-01-11T21:44:22.6617989Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6618101Z Ran 1 test in 5.343s 2023-01-11T21:44:22.6618121Z 2023-01-11T21:44:22.6618195Z OK 2023-01-11T21:44:22.6618230Z 2023-01-11T21:44:22.6618335Z Generating XML reports... 2023-01-11T21:44:22.6618781Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212447.xml 2023-01-11T21:44:22.6619148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6619321Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6619695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6619882Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6619905Z 2023-01-11T21:44:22.6620011Z Running tests... 2023-01-11T21:44:22.6620273Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6620567Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6620966Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip: Requires file:// initialization method. Both tcp:// and env:// rely on the TCP store for which reinitialization has proven racy. (0.002s) 2023-01-11T21:44:22.6620989Z 2023-01-11T21:44:22.6621252Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6621362Z Ran 1 test in 0.002s 2023-01-11T21:44:22.6621381Z 2023-01-11T21:44:22.6621487Z OK (skipped=1) 2023-01-11T21:44:22.6621507Z 2023-01-11T21:44:22.6621627Z Generating XML reports... 2023-01-11T21:44:22.6622070Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212454.xml 2023-01-11T21:44:22.6622496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6622668Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6623027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6623212Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6623231Z 2023-01-11T21:44:22.6623340Z Running tests... 2023-01-11T21:44:22.6623603Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6623911Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6624169Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6624385Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11745 2023-01-11T21:44:22.6624601Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11746 2023-01-11T21:44:22.6624947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6625120Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6625491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6625724Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6626089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6626260Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6626626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6626813Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6627054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6627431Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6627671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6628058Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6628283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6628508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6628665Z skip: Skipped due to small world size. (4.337s) 2023-01-11T21:44:22.6628684Z 2023-01-11T21:44:22.6628947Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6629060Z Ran 1 test in 4.337s 2023-01-11T21:44:22.6629080Z 2023-01-11T21:44:22.6629185Z OK (skipped=1) 2023-01-11T21:44:22.6629205Z 2023-01-11T21:44:22.6629309Z Generating XML reports... 2023-01-11T21:44:22.6629753Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212457.xml 2023-01-11T21:44:22.6630118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6630292Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6630667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6630855Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6630874Z 2023-01-11T21:44:22.6630981Z Running tests... 2023-01-11T21:44:22.6631243Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6631590Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6631852Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6632068Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11854 2023-01-11T21:44:22.6632285Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11855 2023-01-11T21:44:22.6632651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6632823Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6633200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6633388Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6633752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6633905Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6634271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6634453Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6634740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6634985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6635381Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6635769Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6635997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6636222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6636305Z ok (4.249s) 2023-01-11T21:44:22.6636325Z 2023-01-11T21:44:22.6636589Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6636699Z Ran 1 test in 4.249s 2023-01-11T21:44:22.6636718Z 2023-01-11T21:44:22.6636814Z OK 2023-01-11T21:44:22.6636833Z 2023-01-11T21:44:22.6636955Z Generating XML reports... 2023-01-11T21:44:22.6637397Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212504.xml 2023-01-11T21:44:22.6637761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6637933Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6638296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6638484Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6638503Z 2023-01-11T21:44:22.6638610Z Running tests... 2023-01-11T21:44:22.6638869Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6639182Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6639448Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6639663Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11963 2023-01-11T21:44:22.6639877Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11964 2023-01-11T21:44:22.6640241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6640450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6640823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6641010Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6641366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6641539Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6641906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6642092Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6642331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6642552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6642953Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6643346Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6643571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6643838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6643945Z ok (4.211s) 2023-01-11T21:44:22.6643964Z 2023-01-11T21:44:22.6644228Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6644339Z Ran 1 test in 4.212s 2023-01-11T21:44:22.6644358Z 2023-01-11T21:44:22.6644450Z OK 2023-01-11T21:44:22.6644469Z 2023-01-11T21:44:22.6644574Z Generating XML reports... 2023-01-11T21:44:22.6645024Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212510.xml 2023-01-11T21:44:22.6645389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6645559Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6645936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6646123Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6646142Z 2023-01-11T21:44:22.6646251Z Running tests... 2023-01-11T21:44:22.6646512Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6646804Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6647075Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2023-01-11T21:44:22.6647098Z 2023-01-11T21:44:22.6647362Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6647471Z Ran 1 test in 0.002s 2023-01-11T21:44:22.6647491Z 2023-01-11T21:44:22.6647597Z OK (skipped=1) 2023-01-11T21:44:22.6647616Z 2023-01-11T21:44:22.6647737Z Generating XML reports... 2023-01-11T21:44:22.6648182Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212517.xml 2023-01-11T21:44:22.6648547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6648720Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6649076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6649262Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6649329Z 2023-01-11T21:44:22.6649441Z Running tests... 2023-01-11T21:44:22.6649702Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6650010Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6650263Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s) 2023-01-11T21:44:22.6650448Z 2023-01-11T21:44:22.6650732Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6650845Z Ran 1 test in 0.003s 2023-01-11T21:44:22.6650865Z 2023-01-11T21:44:22.6650971Z OK (skipped=1) 2023-01-11T21:44:22.6650990Z 2023-01-11T21:44:22.6651097Z Generating XML reports... 2023-01-11T21:44:22.6651543Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212519.xml 2023-01-11T21:44:22.6651908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6652088Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6652463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6652650Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6652669Z 2023-01-11T21:44:22.6652776Z Running tests... 2023-01-11T21:44:22.6653162Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6653486Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6653733Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2023-01-11T21:44:22.6653754Z 2023-01-11T21:44:22.6654012Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6654122Z Ran 1 test in 0.002s 2023-01-11T21:44:22.6654145Z 2023-01-11T21:44:22.6654250Z OK (skipped=1) 2023-01-11T21:44:22.6654269Z 2023-01-11T21:44:22.6654389Z Generating XML reports... 2023-01-11T21:44:22.6654832Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212522.xml 2023-01-11T21:44:22.6655193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6655367Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6655721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6655910Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6655929Z 2023-01-11T21:44:22.6656036Z Running tests... 2023-01-11T21:44:22.6656297Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6656608Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6656859Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2023-01-11T21:44:22.6656878Z 2023-01-11T21:44:22.6657138Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6657249Z Ran 1 test in 0.002s 2023-01-11T21:44:22.6657268Z 2023-01-11T21:44:22.6657374Z OK (skipped=1) 2023-01-11T21:44:22.6657392Z 2023-01-11T21:44:22.6657500Z Generating XML reports... 2023-01-11T21:44:22.6657946Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212524.xml 2023-01-11T21:44:22.6658308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6658481Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6658855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6659112Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6659131Z 2023-01-11T21:44:22.6659237Z Running tests... 2023-01-11T21:44:22.6659501Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6659813Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6660060Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2023-01-11T21:44:22.6660079Z 2023-01-11T21:44:22.6660338Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6660450Z Ran 1 test in 0.002s 2023-01-11T21:44:22.6660469Z 2023-01-11T21:44:22.6660575Z OK (skipped=1) 2023-01-11T21:44:22.6660594Z 2023-01-11T21:44:22.6660716Z Generating XML reports... 2023-01-11T21:44:22.6661155Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212527.xml 2023-01-11T21:44:22.6661522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6661693Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6662062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6662279Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6662299Z 2023-01-11T21:44:22.6662414Z Running tests... 2023-01-11T21:44:22.6662677Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6662983Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6663255Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2023-01-11T21:44:22.6663277Z 2023-01-11T21:44:22.6663537Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6663648Z Ran 1 test in 0.002s 2023-01-11T21:44:22.6663667Z 2023-01-11T21:44:22.6663772Z OK (skipped=1) 2023-01-11T21:44:22.6663792Z 2023-01-11T21:44:22.6663896Z Generating XML reports... 2023-01-11T21:44:22.6664340Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212529.xml 2023-01-11T21:44:22.6664709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6664880Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6665255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6665441Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6665463Z 2023-01-11T21:44:22.6665570Z Running tests... 2023-01-11T21:44:22.6665832Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6666140Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6666380Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2023-01-11T21:44:22.6666416Z 2023-01-11T21:44:22.6666661Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6666772Z Ran 1 test in 0.002s 2023-01-11T21:44:22.6666791Z 2023-01-11T21:44:22.6666899Z OK (skipped=1) 2023-01-11T21:44:22.6666932Z 2023-01-11T21:44:22.6667048Z Generating XML reports... 2023-01-11T21:44:22.6667481Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212531.xml 2023-01-11T21:44:22.6667842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6668082Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6668457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6668627Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6668646Z 2023-01-11T21:44:22.6668753Z Running tests... 2023-01-11T21:44:22.6669018Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6669324Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6669582Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2023-01-11T21:44:22.6669613Z 2023-01-11T21:44:22.6669862Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6669972Z Ran 1 test in 0.002s 2023-01-11T21:44:22.6669995Z 2023-01-11T21:44:22.6670101Z OK (skipped=1) 2023-01-11T21:44:22.6670120Z 2023-01-11T21:44:22.6670242Z Generating XML reports... 2023-01-11T21:44:22.6670659Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212534.xml 2023-01-11T21:44:22.6671022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6671237Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6671618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6671805Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6671824Z 2023-01-11T21:44:22.6671930Z Running tests... 2023-01-11T21:44:22.6672198Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6672500Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6672727Z test_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6672944Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12336 2023-01-11T21:44:22.6673154Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12337 2023-01-11T21:44:22.6673563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6673737Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6674108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6674295Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6674656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6674829Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6675176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6675360Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6675603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6675847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6676245Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6676632Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6676857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6677141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6677460Z STAGE:2023-01-11 21:25:40 12336:12336 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6677780Z STAGE:2023-01-11 21:25:40 12337:12337 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6678112Z STAGE:2023-01-11 21:25:40 12336:12336 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6678449Z STAGE:2023-01-11 21:25:40 12337:12337 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6678785Z STAGE:2023-01-11 21:25:40 12336:12336 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6679139Z STAGE:2023-01-11 21:25:40 12337:12337 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6679447Z STAGE:2023-01-11 21:25:40 12336:12336 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6679773Z STAGE:2023-01-11 21:25:40 12337:12337 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6680101Z STAGE:2023-01-11 21:25:40 12337:12337 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6680402Z STAGE:2023-01-11 21:25:40 12336:12336 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6680787Z STAGE:2023-01-11 21:25:40 12337:12337 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6681141Z STAGE:2023-01-11 21:25:40 12336:12336 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6681463Z STAGE:2023-01-11 21:25:40 12337:12337 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6681781Z STAGE:2023-01-11 21:25:40 12336:12336 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6682110Z STAGE:2023-01-11 21:25:40 12336:12336 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6682452Z STAGE:2023-01-11 21:25:40 12336:12336 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6682779Z STAGE:2023-01-11 21:25:40 12337:12337 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6683115Z STAGE:2023-01-11 21:25:40 12337:12337 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6683423Z STAGE:2023-01-11 21:25:40 12336:12336 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6683738Z STAGE:2023-01-11 21:25:40 12337:12337 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6684063Z STAGE:2023-01-11 21:25:40 12337:12337 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6684380Z STAGE:2023-01-11 21:25:40 12336:12336 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6684720Z STAGE:2023-01-11 21:25:40 12337:12337 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6685059Z STAGE:2023-01-11 21:25:40 12336:12336 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6685377Z STAGE:2023-01-11 21:25:40 12337:12337 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6685689Z STAGE:2023-01-11 21:25:40 12336:12336 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6686016Z STAGE:2023-01-11 21:25:40 12336:12336 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6686334Z STAGE:2023-01-11 21:25:40 12336:12336 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6686657Z STAGE:2023-01-11 21:25:40 12337:12337 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6686993Z STAGE:2023-01-11 21:25:40 12337:12337 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6687390Z STAGE:2023-01-11 21:25:40 12336:12336 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6687703Z STAGE:2023-01-11 21:25:40 12337:12337 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6688027Z STAGE:2023-01-11 21:25:40 12337:12337 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6688346Z STAGE:2023-01-11 21:25:40 12336:12336 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6688687Z STAGE:2023-01-11 21:25:40 12337:12337 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6689024Z STAGE:2023-01-11 21:25:40 12336:12336 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6689325Z STAGE:2023-01-11 21:25:40 12337:12337 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6689647Z STAGE:2023-01-11 21:25:40 12336:12336 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6689976Z STAGE:2023-01-11 21:25:40 12336:12336 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6690704Z STAGE:2023-01-11 21:25:40 12336:12336 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:25:40 12337:12337 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6690725Z 2023-01-11T21:44:22.6691143Z STAGE:2023-01-11 21:25:40 12337:12337 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6691479Z STAGE:2023-01-11 21:25:40 12336:12336 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6691797Z STAGE:2023-01-11 21:25:40 12337:12337 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6692123Z STAGE:2023-01-11 21:25:40 12337:12337 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6692442Z STAGE:2023-01-11 21:25:40 12336:12336 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6692781Z STAGE:2023-01-11 21:25:40 12337:12337 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6693101Z STAGE:2023-01-11 21:25:40 12336:12336 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6693420Z STAGE:2023-01-11 21:25:40 12337:12337 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6693738Z STAGE:2023-01-11 21:25:40 12336:12336 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6694066Z STAGE:2023-01-11 21:25:40 12336:12336 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6694615Z STAGE:2023-01-11 21:25:40 12336:12336 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:25:40 12337:12337 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6694635Z 2023-01-11T21:44:22.6694968Z STAGE:2023-01-11 21:25:40 12337:12337 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6695288Z STAGE:2023-01-11 21:25:40 12336:12336 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6695598Z STAGE:2023-01-11 21:25:40 12337:12337 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6695920Z STAGE:2023-01-11 21:25:40 12337:12337 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6696241Z STAGE:2023-01-11 21:25:40 12336:12336 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6696563Z STAGE:2023-01-11 21:25:40 12337:12337 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6696899Z STAGE:2023-01-11 21:25:40 12336:12336 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6697217Z STAGE:2023-01-11 21:25:40 12337:12337 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6697612Z STAGE:2023-01-11 21:25:40 12336:12336 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6697938Z STAGE:2023-01-11 21:25:40 12336:12336 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6698275Z STAGE:2023-01-11 21:25:40 12336:12336 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6698596Z STAGE:2023-01-11 21:25:40 12337:12337 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6698936Z STAGE:2023-01-11 21:25:40 12337:12337 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6699257Z STAGE:2023-01-11 21:25:40 12336:12336 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6699557Z STAGE:2023-01-11 21:25:40 12337:12337 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6699884Z STAGE:2023-01-11 21:25:40 12337:12337 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6700211Z STAGE:2023-01-11 21:25:40 12336:12336 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6700549Z STAGE:2023-01-11 21:25:40 12337:12337 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6700884Z STAGE:2023-01-11 21:25:40 12336:12336 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6700985Z ok (4.224s) 2023-01-11T21:44:22.6701006Z 2023-01-11T21:44:22.6701318Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6701435Z Ran 1 test in 4.224s 2023-01-11T21:44:22.6701455Z 2023-01-11T21:44:22.6701528Z OK 2023-01-11T21:44:22.6701564Z 2023-01-11T21:44:22.6701669Z Generating XML reports... 2023-01-11T21:44:22.6702124Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212536.xml 2023-01-11T21:44:22.6702489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6702671Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6703044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6703230Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6703250Z 2023-01-11T21:44:22.6703357Z Running tests... 2023-01-11T21:44:22.6703625Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6703918Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6704167Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6704915Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.596s) 2023-01-11T21:44:22.6704939Z 2023-01-11T21:44:22.6705201Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6705311Z Ran 1 test in 1.596s 2023-01-11T21:44:22.6705330Z 2023-01-11T21:44:22.6705435Z OK (skipped=1) 2023-01-11T21:44:22.6705454Z 2023-01-11T21:44:22.6705576Z Generating XML reports... 2023-01-11T21:44:22.6706023Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212543.xml 2023-01-11T21:44:22.6706385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6706559Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6706917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6707162Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6707181Z 2023-01-11T21:44:22.6707293Z Running tests... 2023-01-11T21:44:22.6707559Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6707867Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6708126Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6708347Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12483 2023-01-11T21:44:22.6708561Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12484 2023-01-11T21:44:22.6708906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6709077Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6709449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6709638Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6709996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6710164Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6710576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6710765Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6711008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6711229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6711630Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6712024Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6712251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6712476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6712714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T21:44:22.6712949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T21:44:22.6713343Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.6713671Z STAGE:2023-01-11 21:25:51 12484:12484 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6714048Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.6714372Z STAGE:2023-01-11 21:25:51 12483:12483 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6714701Z STAGE:2023-01-11 21:25:51 12483:12483 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6715023Z STAGE:2023-01-11 21:25:51 12484:12484 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6715366Z STAGE:2023-01-11 21:25:51 12483:12483 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6715706Z STAGE:2023-01-11 21:25:51 12484:12484 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6716025Z STAGE:2023-01-11 21:25:51 12483:12483 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6716338Z STAGE:2023-01-11 21:25:51 12484:12484 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6716723Z STAGE:2023-01-11 21:25:51 12484:12484 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6717043Z STAGE:2023-01-11 21:25:51 12484:12484 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6717373Z STAGE:2023-01-11 21:25:51 12483:12483 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6717711Z STAGE:2023-01-11 21:25:51 12483:12483 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6718036Z STAGE:2023-01-11 21:25:51 12484:12484 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6718350Z STAGE:2023-01-11 21:25:51 12483:12483 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6718675Z STAGE:2023-01-11 21:25:51 12483:12483 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6718997Z STAGE:2023-01-11 21:25:51 12484:12484 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6719341Z STAGE:2023-01-11 21:25:51 12483:12483 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6719675Z STAGE:2023-01-11 21:25:51 12484:12484 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6719988Z STAGE:2023-01-11 21:25:51 12483:12483 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6720335Z STAGE:2023-01-11 21:25:51 12484:12484 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6720666Z STAGE:2023-01-11 21:25:51 12484:12484 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6721000Z STAGE:2023-01-11 21:25:51 12484:12484 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6721324Z STAGE:2023-01-11 21:25:51 12483:12483 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6721661Z STAGE:2023-01-11 21:25:51 12483:12483 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6721987Z STAGE:2023-01-11 21:25:51 12484:12484 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6722300Z STAGE:2023-01-11 21:25:51 12483:12483 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6722627Z STAGE:2023-01-11 21:25:51 12483:12483 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6722935Z STAGE:2023-01-11 21:25:51 12484:12484 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6723273Z STAGE:2023-01-11 21:25:51 12483:12483 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6723610Z STAGE:2023-01-11 21:25:51 12484:12484 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6723929Z STAGE:2023-01-11 21:25:51 12483:12483 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6724254Z STAGE:2023-01-11 21:25:51 12484:12484 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6724580Z STAGE:2023-01-11 21:25:51 12484:12484 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6724916Z STAGE:2023-01-11 21:25:51 12484:12484 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6725244Z STAGE:2023-01-11 21:25:51 12483:12483 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6725583Z STAGE:2023-01-11 21:25:51 12483:12483 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6725886Z STAGE:2023-01-11 21:25:51 12484:12484 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6726198Z STAGE:2023-01-11 21:25:51 12483:12483 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6726521Z STAGE:2023-01-11 21:25:51 12483:12483 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6726898Z STAGE:2023-01-11 21:25:51 12484:12484 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6727233Z STAGE:2023-01-11 21:25:51 12483:12483 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6727568Z STAGE:2023-01-11 21:25:51 12484:12484 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6727893Z STAGE:2023-01-11 21:25:51 12483:12483 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6728208Z STAGE:2023-01-11 21:25:51 12484:12484 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6728515Z STAGE:2023-01-11 21:25:51 12484:12484 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6728850Z STAGE:2023-01-11 21:25:51 12484:12484 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6729172Z STAGE:2023-01-11 21:25:51 12483:12483 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6729510Z STAGE:2023-01-11 21:25:51 12483:12483 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6729829Z STAGE:2023-01-11 21:25:51 12484:12484 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6730145Z STAGE:2023-01-11 21:25:51 12483:12483 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6730993Z STAGE:2023-01-11 21:25:51 12484:12484 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:25:51 12483:12483 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6731014Z 2023-01-11T21:44:22.6731580Z STAGE:2023-01-11 21:25:51 12484:12484 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:25:51 12483:12483 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6731600Z 2023-01-11T21:44:22.6731921Z STAGE:2023-01-11 21:25:51 12483:12483 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6732240Z STAGE:2023-01-11 21:25:51 12484:12484 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6732567Z STAGE:2023-01-11 21:25:51 12484:12484 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6732889Z STAGE:2023-01-11 21:25:51 12484:12484 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6733217Z STAGE:2023-01-11 21:25:51 12483:12483 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6733553Z STAGE:2023-01-11 21:25:51 12483:12483 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6733871Z STAGE:2023-01-11 21:25:51 12484:12484 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6734186Z STAGE:2023-01-11 21:25:51 12483:12483 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6734511Z STAGE:2023-01-11 21:25:51 12483:12483 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6734836Z STAGE:2023-01-11 21:25:51 12484:12484 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6735174Z STAGE:2023-01-11 21:25:51 12483:12483 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6735511Z STAGE:2023-01-11 21:25:51 12484:12484 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6735819Z STAGE:2023-01-11 21:25:51 12483:12483 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6736141Z STAGE:2023-01-11 21:25:51 12484:12484 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.6736466Z STAGE:2023-01-11 21:25:51 12484:12484 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6736801Z STAGE:2023-01-11 21:25:51 12484:12484 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6737210Z STAGE:2023-01-11 21:25:51 12483:12483 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.6737550Z STAGE:2023-01-11 21:25:51 12483:12483 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.6737652Z ok (4.287s) 2023-01-11T21:44:22.6737672Z 2023-01-11T21:44:22.6737939Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6738051Z Ran 1 test in 4.288s 2023-01-11T21:44:22.6738071Z 2023-01-11T21:44:22.6738148Z OK 2023-01-11T21:44:22.6738167Z 2023-01-11T21:44:22.6738290Z Generating XML reports... 2023-01-11T21:44:22.6738737Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212547.xml 2023-01-11T21:44:22.6739103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6739276Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6739658Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6739848Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6739867Z 2023-01-11T21:44:22.6739974Z Running tests... 2023-01-11T21:44:22.6740220Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6740580Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6740841Z test_broadcast_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6741059Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12602 2023-01-11T21:44:22.6741274Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12603 2023-01-11T21:44:22.6741639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6741816Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6742189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6742379Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6742722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6742894Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6743260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6743448Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6743688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6743927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6744327Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6744715Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6744941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6745149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6745307Z skip: Skipped due to small world size. (4.120s) 2023-01-11T21:44:22.6745327Z 2023-01-11T21:44:22.6745592Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6745703Z Ran 1 test in 4.120s 2023-01-11T21:44:22.6745723Z 2023-01-11T21:44:22.6745829Z OK (skipped=1) 2023-01-11T21:44:22.6745848Z 2023-01-11T21:44:22.6746022Z Generating XML reports... 2023-01-11T21:44:22.6746471Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212554.xml 2023-01-11T21:44:22.6746837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6746991Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6747368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6747553Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6747572Z 2023-01-11T21:44:22.6747678Z Running tests... 2023-01-11T21:44:22.6747940Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6748249Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6748505Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6748725Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12711 2023-01-11T21:44:22.6748938Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12712 2023-01-11T21:44:22.6749287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6749520Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6749910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6750095Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6750454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6750624Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6750993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6751178Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6751402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6751640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6752034Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6752420Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6752643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6752866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6753636Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1478: UserWarning: torch.distributed.broadcast_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2023-01-11T21:44:22.6753748Z warnings.warn( 2023-01-11T21:44:22.6754505Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1478: UserWarning: torch.distributed.broadcast_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2023-01-11T21:44:22.6754614Z warnings.warn( 2023-01-11T21:44:22.6754697Z ok (5.749s) 2023-01-11T21:44:22.6754716Z 2023-01-11T21:44:22.6754984Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6755095Z Ran 1 test in 5.749s 2023-01-11T21:44:22.6755115Z 2023-01-11T21:44:22.6755258Z OK 2023-01-11T21:44:22.6755277Z 2023-01-11T21:44:22.6755403Z Generating XML reports... 2023-01-11T21:44:22.6755850Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212601.xml 2023-01-11T21:44:22.6756216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6756389Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6756745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6756934Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6756953Z 2023-01-11T21:44:22.6757061Z Running tests... 2023-01-11T21:44:22.6757323Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6757635Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6757897Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6758637Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.604s) 2023-01-11T21:44:22.6758657Z 2023-01-11T21:44:22.6758968Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6759083Z Ran 1 test in 1.604s 2023-01-11T21:44:22.6759102Z 2023-01-11T21:44:22.6759207Z OK (skipped=1) 2023-01-11T21:44:22.6759226Z 2023-01-11T21:44:22.6759331Z Generating XML reports... 2023-01-11T21:44:22.6759780Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212609.xml 2023-01-11T21:44:22.6760147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6760318Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6760697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6760886Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6760905Z 2023-01-11T21:44:22.6761015Z Running tests... 2023-01-11T21:44:22.6761281Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6761592Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6761882Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6762617Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85012 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.613s) 2023-01-11T21:44:22.6762640Z 2023-01-11T21:44:22.6762906Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6763017Z Ran 1 test in 1.613s 2023-01-11T21:44:22.6763036Z 2023-01-11T21:44:22.6763142Z OK (skipped=1) 2023-01-11T21:44:22.6763161Z 2023-01-11T21:44:22.6763286Z Generating XML reports... 2023-01-11T21:44:22.6763730Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212613.xml 2023-01-11T21:44:22.6764094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6764267Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6764622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6764865Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6764885Z 2023-01-11T21:44:22.6764993Z Running tests... 2023-01-11T21:44:22.6765257Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6765567Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6765881Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6766613Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85339 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.589s) 2023-01-11T21:44:22.6766636Z 2023-01-11T21:44:22.6766900Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6779192Z Ran 1 test in 1.589s 2023-01-11T21:44:22.6779224Z 2023-01-11T21:44:22.6779387Z OK (skipped=1) 2023-01-11T21:44:22.6779407Z 2023-01-11T21:44:22.6779519Z Generating XML reports... 2023-01-11T21:44:22.6780023Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212617.xml 2023-01-11T21:44:22.6780543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6780734Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6781118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6781311Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6781331Z 2023-01-11T21:44:22.6781445Z Running tests... 2023-01-11T21:44:22.6781715Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6782028Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6782280Z test_ddp_apply_optim_in_backward (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6782496Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12924 2023-01-11T21:44:22.6782713Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12925 2023-01-11T21:44:22.6783083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6783255Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6783628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6783818Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6784182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6784335Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6784705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6784892Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6785139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6785383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6785782Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6786176Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6786483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6786709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6787487Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired. 2023-01-11T21:44:22.6787584Z warnings.warn( 2023-01-11T21:44:22.6788338Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired. 2023-01-11T21:44:22.6788451Z warnings.warn( 2023-01-11T21:44:22.6788686Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6788915Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6789145Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6789433Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6789667Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6789892Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6790101Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6790326Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6790431Z ok (7.727s) 2023-01-11T21:44:22.6790452Z 2023-01-11T21:44:22.6790726Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6790841Z Ran 1 test in 7.727s 2023-01-11T21:44:22.6790860Z 2023-01-11T21:44:22.6790952Z OK 2023-01-11T21:44:22.6790971Z 2023-01-11T21:44:22.6791093Z Generating XML reports... 2023-01-11T21:44:22.6791547Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212621.xml 2023-01-11T21:44:22.6791894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6792068Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6792444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6792630Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6792652Z 2023-01-11T21:44:22.6792761Z Running tests... 2023-01-11T21:44:22.6793024Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6793338Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6793639Z test_ddp_apply_optim_in_backward_grad_as_bucket_view_false (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6793857Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13039 2023-01-11T21:44:22.6794054Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13040 2023-01-11T21:44:22.6794421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6794596Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6794972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6795220Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6795586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6795759Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6796131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6796297Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6796541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6796779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6797174Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6797569Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6797794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6798018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6798831Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired. 2023-01-11T21:44:22.6798950Z warnings.warn( 2023-01-11T21:44:22.6799715Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired. 2023-01-11T21:44:22.6799826Z warnings.warn( 2023-01-11T21:44:22.6800042Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6800267Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6800500Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6800731Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6800832Z ok (6.821s) 2023-01-11T21:44:22.6800853Z 2023-01-11T21:44:22.6801123Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6801235Z Ran 1 test in 6.822s 2023-01-11T21:44:22.6801254Z 2023-01-11T21:44:22.6801345Z OK 2023-01-11T21:44:22.6801364Z 2023-01-11T21:44:22.6801469Z Generating XML reports... 2023-01-11T21:44:22.6801925Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212631.xml 2023-01-11T21:44:22.6802289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6802464Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6802841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6803028Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6803048Z 2023-01-11T21:44:22.6803156Z Running tests... 2023-01-11T21:44:22.6803420Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6803711Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6804000Z test_ddp_apply_optim_in_backward_ignored_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6804276Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13154 2023-01-11T21:44:22.6804488Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13155 2023-01-11T21:44:22.6804854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6805026Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6805396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6805584Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6805950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6806103Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6806473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6806656Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6806900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6807140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6807579Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6807979Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6808206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6808431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6809184Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired. 2023-01-11T21:44:22.6809298Z warnings.warn( 2023-01-11T21:44:22.6810059Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired. 2023-01-11T21:44:22.6810168Z warnings.warn( 2023-01-11T21:44:22.6810271Z ok (7.405s) 2023-01-11T21:44:22.6810559Z 2023-01-11T21:44:22.6810855Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6810975Z Ran 1 test in 7.405s 2023-01-11T21:44:22.6810996Z 2023-01-11T21:44:22.6811090Z OK 2023-01-11T21:44:22.6811109Z 2023-01-11T21:44:22.6811234Z Generating XML reports... 2023-01-11T21:44:22.6811664Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212641.xml 2023-01-11T21:44:22.6812027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6812204Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6812579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6812768Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6812788Z 2023-01-11T21:44:22.6812897Z Running tests... 2023-01-11T21:44:22.6813166Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6813642Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6813903Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6814101Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13271 2023-01-11T21:44:22.6814315Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13272 2023-01-11T21:44:22.6814683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6814854Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6815229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6815415Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6815776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6815951Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6816301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6816485Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6816794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6817047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6817443Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6817834Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6818060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6818288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6818522Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6818735Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6818839Z ok (6.226s) 2023-01-11T21:44:22.6818859Z 2023-01-11T21:44:22.6819128Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6819238Z Ran 1 test in 6.227s 2023-01-11T21:44:22.6819257Z 2023-01-11T21:44:22.6819346Z OK 2023-01-11T21:44:22.6819365Z 2023-01-11T21:44:22.6819489Z Generating XML reports... 2023-01-11T21:44:22.6819938Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212651.xml 2023-01-11T21:44:22.6820303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6820481Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6820838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6821026Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6821045Z 2023-01-11T21:44:22.6821153Z Running tests... 2023-01-11T21:44:22.6821420Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6821731Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6822001Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6822216Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13386 2023-01-11T21:44:22.6822429Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13387 2023-01-11T21:44:22.6822836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6823012Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6823385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6823577Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6823937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6824110Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6824476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6824662Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6824907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6825128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6825525Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6825963Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6826196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6826421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6826655Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6826887Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6827117Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6827327Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6827428Z ok (6.111s) 2023-01-11T21:44:22.6827448Z 2023-01-11T21:44:22.6827722Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6827834Z Ran 1 test in 6.111s 2023-01-11T21:44:22.6827854Z 2023-01-11T21:44:22.6827949Z OK 2023-01-11T21:44:22.6827968Z 2023-01-11T21:44:22.6828091Z Generating XML reports... 2023-01-11T21:44:22.6828542Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212700.xml 2023-01-11T21:44:22.6828903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6829075Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6829436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6829624Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6829644Z 2023-01-11T21:44:22.6829751Z Running tests... 2023-01-11T21:44:22.6830016Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6830328Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6830594Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6831342Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78641 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.614s) 2023-01-11T21:44:22.6831413Z 2023-01-11T21:44:22.6831684Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6831796Z Ran 1 test in 1.614s 2023-01-11T21:44:22.6831815Z 2023-01-11T21:44:22.6831905Z OK (skipped=1) 2023-01-11T21:44:22.6831942Z 2023-01-11T21:44:22.6832047Z Generating XML reports... 2023-01-11T21:44:22.6832495Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212708.xml 2023-01-11T21:44:22.6832866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6833041Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6833419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6833608Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6833628Z 2023-01-11T21:44:22.6833739Z Running tests... 2023-01-11T21:44:22.6834005Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6834296Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6834579Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6835370Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.601s) 2023-01-11T21:44:22.6835392Z 2023-01-11T21:44:22.6835659Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6835771Z Ran 1 test in 1.602s 2023-01-11T21:44:22.6835790Z 2023-01-11T21:44:22.6835897Z OK (skipped=1) 2023-01-11T21:44:22.6835916Z 2023-01-11T21:44:22.6836043Z Generating XML reports... 2023-01-11T21:44:22.6836487Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212712.xml 2023-01-11T21:44:22.6836854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6837026Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6837384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6837575Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6837593Z 2023-01-11T21:44:22.6837701Z Running tests... 2023-01-11T21:44:22.6837965Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6838275Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6838556Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6838776Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13569 2023-01-11T21:44:22.6838990Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13570 2023-01-11T21:44:22.6839337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6839513Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6839891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6840077Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6840438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6840605Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6841054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6841238Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6841480Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6841862Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6842098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6842482Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6842709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6842930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6843142Z 2023-01-11T21:44:22.6843243Z ok (5.709s) 2023-01-11T21:44:22.6843263Z 2023-01-11T21:44:22.6843526Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6843619Z Ran 1 test in 5.709s 2023-01-11T21:44:22.6843655Z 2023-01-11T21:44:22.6843729Z OK 2023-01-11T21:44:22.6843748Z 2023-01-11T21:44:22.6843870Z Generating XML reports... 2023-01-11T21:44:22.6844360Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212716.xml 2023-01-11T21:44:22.6844737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6844912Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6845287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6845478Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6845498Z 2023-01-11T21:44:22.6845607Z Running tests... 2023-01-11T21:44:22.6845851Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6846161Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6846465Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6846681Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13680 2023-01-11T21:44:22.6846893Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13681 2023-01-11T21:44:22.6847261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6847432Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6847809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6847978Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6848339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6848508Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6848881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6849067Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6849307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6849545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6849996Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6850570Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6850787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6851019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6851123Z ok (5.816s) 2023-01-11T21:44:22.6851142Z 2023-01-11T21:44:22.6851413Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6851526Z Ran 1 test in 5.816s 2023-01-11T21:44:22.6851546Z 2023-01-11T21:44:22.6851638Z OK 2023-01-11T21:44:22.6851657Z 2023-01-11T21:44:22.6851782Z Generating XML reports... 2023-01-11T21:44:22.6852230Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212725.xml 2023-01-11T21:44:22.6852602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6852759Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6853136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6853325Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6853420Z 2023-01-11T21:44:22.6853536Z Running tests... 2023-01-11T21:44:22.6853804Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6854113Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6854373Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6854589Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13791 2023-01-11T21:44:22.6854787Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13792 2023-01-11T21:44:22.6855150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6855324Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6855702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6855889Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6856246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6856415Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6856781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6856968Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6857192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6857432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6857827Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6858223Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6858449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6858673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6858909Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6859217Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6859301Z ok (6.109s) 2023-01-11T21:44:22.6859339Z 2023-01-11T21:44:22.6859590Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6859704Z Ran 1 test in 6.109s 2023-01-11T21:44:22.6859723Z 2023-01-11T21:44:22.6859815Z OK 2023-01-11T21:44:22.6859834Z 2023-01-11T21:44:22.6859959Z Generating XML reports... 2023-01-11T21:44:22.6860410Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212733.xml 2023-01-11T21:44:22.6860780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6860959Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6861334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6861508Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6861527Z 2023-01-11T21:44:22.6861636Z Running tests... 2023-01-11T21:44:22.6861897Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6862206Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6862537Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6862761Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13906 2023-01-11T21:44:22.6862977Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13907 2023-01-11T21:44:22.6863345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6863498Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6863878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6864065Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6864422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6864594Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6864963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6865148Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6865391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6865632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6866008Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6866402Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6866628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6866852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6867621Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-01-11T21:44:22.6867777Z ok (6.267s) 2023-01-11T21:44:22.6867797Z 2023-01-11T21:44:22.6868066Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6868180Z Ran 1 test in 6.267s 2023-01-11T21:44:22.6868199Z 2023-01-11T21:44:22.6868292Z OK 2023-01-11T21:44:22.6868311Z 2023-01-11T21:44:22.6868434Z Generating XML reports... 2023-01-11T21:44:22.6868864Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212742.xml 2023-01-11T21:44:22.6869232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6869405Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6869783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6869972Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6869995Z 2023-01-11T21:44:22.6870105Z Running tests... 2023-01-11T21:44:22.6870367Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6870679Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6870954Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6871743Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78235 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.648s) 2023-01-11T21:44:22.6871765Z 2023-01-11T21:44:22.6872016Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6872129Z Ran 1 test in 1.648s 2023-01-11T21:44:22.6872147Z 2023-01-11T21:44:22.6872258Z OK (skipped=1) 2023-01-11T21:44:22.6872277Z 2023-01-11T21:44:22.6872401Z Generating XML reports... 2023-01-11T21:44:22.6872848Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212750.xml 2023-01-11T21:44:22.6873213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6873387Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6873809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6873998Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6874017Z 2023-01-11T21:44:22.6874107Z Running tests... 2023-01-11T21:44:22.6874374Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6874682Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6874942Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6875157Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14055 2023-01-11T21:44:22.6875371Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14056 2023-01-11T21:44:22.6875740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6875913Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6876271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6876461Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6876827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6877070Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6877442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6877629Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6877873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6878115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6878512Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6878885Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6879111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6879339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6880224Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2023-01-11T21:44:22.6881151Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2023-01-11T21:44:22.6882320Z /opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py:197: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1134.) 2023-01-11T21:44:22.6882558Z Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2023-01-11T21:44:22.6883711Z /opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py:197: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1134.) 2023-01-11T21:44:22.6883940Z Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2023-01-11T21:44:22.6884175Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6884407Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6885284Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2023-01-11T21:44:22.6886158Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2023-01-11T21:44:22.6887082Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2023-01-11T21:44:22.6887941Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2023-01-11T21:44:22.6888812Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2023-01-11T21:44:22.6889711Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2023-01-11T21:44:22.6890752Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2023-01-11T21:44:22.6891644Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2023-01-11T21:44:22.6892511Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2023-01-11T21:44:22.6893369Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2023-01-11T21:44:22.6893475Z ok (4.246s) 2023-01-11T21:44:22.6893496Z 2023-01-11T21:44:22.6893764Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6893880Z Ran 1 test in 4.246s 2023-01-11T21:44:22.6893900Z 2023-01-11T21:44:22.6893991Z OK 2023-01-11T21:44:22.6894010Z 2023-01-11T21:44:22.6894135Z Generating XML reports... 2023-01-11T21:44:22.6894566Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212755.xml 2023-01-11T21:44:22.6894933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6895192Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6895575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6895765Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6895785Z 2023-01-11T21:44:22.6895894Z Running tests... 2023-01-11T21:44:22.6896161Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6896478Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6896705Z test_ddp_device (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6897444Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77324 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.651s) 2023-01-11T21:44:22.6897487Z 2023-01-11T21:44:22.6897732Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6897845Z Ran 1 test in 1.651s 2023-01-11T21:44:22.6897864Z 2023-01-11T21:44:22.6897971Z OK (skipped=1) 2023-01-11T21:44:22.6897990Z 2023-01-11T21:44:22.6898114Z Generating XML reports... 2023-01-11T21:44:22.6898623Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212801.xml 2023-01-11T21:44:22.6899005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6899179Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6899557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6899728Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6899769Z 2023-01-11T21:44:22.6899859Z Running tests... 2023-01-11T21:44:22.6900123Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6900433Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6900701Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6900922Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14202 2023-01-11T21:44:22.6901141Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14203 2023-01-11T21:44:22.6901507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6901680Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6902040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6902229Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6902589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6902762Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6903130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6903321Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6903564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6903805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6904179Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6904631Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6904859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6905085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6905871Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1331: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior. 2023-01-11T21:44:22.6906204Z warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes " 2023-01-11T21:44:22.6906978Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1331: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior. 2023-01-11T21:44:22.6907306Z warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes " 2023-01-11T21:44:22.6907459Z ok (6.261s) 2023-01-11T21:44:22.6907479Z 2023-01-11T21:44:22.6907750Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6907916Z Ran 1 test in 6.261s 2023-01-11T21:44:22.6907937Z 2023-01-11T21:44:22.6908012Z OK 2023-01-11T21:44:22.6908031Z 2023-01-11T21:44:22.6908156Z Generating XML reports... 2023-01-11T21:44:22.6908603Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212806.xml 2023-01-11T21:44:22.6908970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6909148Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6909523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6909710Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6909729Z 2023-01-11T21:44:22.6909837Z Running tests... 2023-01-11T21:44:22.6910080Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6910394Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6910664Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6911407Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78685 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.589s) 2023-01-11T21:44:22.6911431Z 2023-01-11T21:44:22.6911694Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6911804Z Ran 1 test in 1.589s 2023-01-11T21:44:22.6911823Z 2023-01-11T21:44:22.6911930Z OK (skipped=1) 2023-01-11T21:44:22.6911949Z 2023-01-11T21:44:22.6912074Z Generating XML reports... 2023-01-11T21:44:22.6912519Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212814.xml 2023-01-11T21:44:22.6912887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6913042Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6913419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6913606Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6913673Z 2023-01-11T21:44:22.6913786Z Running tests... 2023-01-11T21:44:22.6914054Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6914362Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6914629Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6915366Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.639s) 2023-01-11T21:44:22.6915387Z 2023-01-11T21:44:22.6915651Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6915763Z Ran 1 test in 1.640s 2023-01-11T21:44:22.6915786Z 2023-01-11T21:44:22.6915875Z OK (skipped=1) 2023-01-11T21:44:22.6915894Z 2023-01-11T21:44:22.6916018Z Generating XML reports... 2023-01-11T21:44:22.6916463Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212819.xml 2023-01-11T21:44:22.6916832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6917049Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6917437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6917628Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6917648Z 2023-01-11T21:44:22.6917756Z Running tests... 2023-01-11T21:44:22.6917999Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6918312Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6918601Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6918817Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14415 2023-01-11T21:44:22.6919031Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14416 2023-01-11T21:44:22.6919400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6919572Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6919945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6920132Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6920472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6920647Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6921015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6921202Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6921448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6921693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6922090Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6922482Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6922709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6922974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6923211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T21:44:22.6923447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T21:44:22.6923847Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.6924235Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.6924468Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6924700Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6924930Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6925162Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6925247Z ok (6.553s) 2023-01-11T21:44:22.6925267Z 2023-01-11T21:44:22.6925534Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6925645Z Ran 1 test in 6.553s 2023-01-11T21:44:22.6925665Z 2023-01-11T21:44:22.6925758Z OK 2023-01-11T21:44:22.6925777Z 2023-01-11T21:44:22.6925946Z Generating XML reports... 2023-01-11T21:44:22.6926404Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212823.xml 2023-01-11T21:44:22.6926770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6926943Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6927297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6927491Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6927510Z 2023-01-11T21:44:22.6927617Z Running tests... 2023-01-11T21:44:22.6927881Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6928189Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6928463Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6928678Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14536 2023-01-11T21:44:22.6928893Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14537 2023-01-11T21:44:22.6929241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6929415Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6929793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6929982Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6930562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6930746Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6931126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6931314Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6931555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6931776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6932264Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6932654Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6932884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6933111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6933385Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2023-01-11T21:44:22.6933656Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2023-01-11T21:44:22.6933887Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6934110Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6934324Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6934553Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6934826Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2023-01-11T21:44:22.6935152Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2023-01-11T21:44:22.6935433Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2023-01-11T21:44:22.6935695Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2023-01-11T21:44:22.6935925Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6936158Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6936389Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6936593Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6936861Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2023-01-11T21:44:22.6937132Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2023-01-11T21:44:22.6937404Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations 2023-01-11T21:44:22.6937667Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations 2023-01-11T21:44:22.6937892Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6938119Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6938349Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6938577Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6938662Z ok (6.948s) 2023-01-11T21:44:22.6938682Z 2023-01-11T21:44:22.6938956Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6939074Z Ran 1 test in 6.948s 2023-01-11T21:44:22.6939094Z 2023-01-11T21:44:22.6939186Z OK 2023-01-11T21:44:22.6939204Z 2023-01-11T21:44:22.6939328Z Generating XML reports... 2023-01-11T21:44:22.6939784Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212832.xml 2023-01-11T21:44:22.6940152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6940327Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6940758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6940949Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6940968Z 2023-01-11T21:44:22.6941078Z Running tests... 2023-01-11T21:44:22.6941342Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6941657Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6941924Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6942664Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77378 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.640s) 2023-01-11T21:44:22.6942687Z 2023-01-11T21:44:22.6942947Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6943059Z Ran 1 test in 1.640s 2023-01-11T21:44:22.6943078Z 2023-01-11T21:44:22.6943186Z OK (skipped=1) 2023-01-11T21:44:22.6943207Z 2023-01-11T21:44:22.6943312Z Generating XML reports... 2023-01-11T21:44:22.6943804Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212841.xml 2023-01-11T21:44:22.6944179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6944353Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6944726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6944917Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6944940Z 2023-01-11T21:44:22.6945050Z Running tests... 2023-01-11T21:44:22.6945319Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6945609Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6945877Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6946097Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14685 2023-01-11T21:44:22.6946314Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14686 2023-01-11T21:44:22.6946682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6946855Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6947231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6947423Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6947782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6947934Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6948304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6948494Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6948735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6948974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6949371Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6949822Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6950047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6950272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6950794Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2023-01-11T21:44:22.6951330Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2023-01-11T21:44:22.6951568Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6951803Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6952073Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations. 2023-01-11T21:44:22.6952437Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations. 2023-01-11T21:44:22.6952738Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created. 2023-01-11T21:44:22.6953032Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created. 2023-01-11T21:44:22.6953357Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0 2023-01-11T21:44:22.6953684Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0 2023-01-11T21:44:22.6954001Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively. 2023-01-11T21:44:22.6954301Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively. 2023-01-11T21:44:22.6954533Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6954764Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6954866Z ok (6.337s) 2023-01-11T21:44:22.6954886Z 2023-01-11T21:44:22.6955163Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6955279Z Ran 1 test in 6.337s 2023-01-11T21:44:22.6955299Z 2023-01-11T21:44:22.6955391Z OK 2023-01-11T21:44:22.6955410Z 2023-01-11T21:44:22.6955533Z Generating XML reports... 2023-01-11T21:44:22.6955980Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212846.xml 2023-01-11T21:44:22.6956333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6956511Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6956890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6957079Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6957098Z 2023-01-11T21:44:22.6957208Z Running tests... 2023-01-11T21:44:22.6957473Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6957843Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6958153Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6958370Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14800 2023-01-11T21:44:22.6958565Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14801 2023-01-11T21:44:22.6958931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6959103Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6959472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6959658Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6960020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6960191Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6960557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6960723Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6961012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6961260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6961659Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6962051Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6962282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6962508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6962739Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6962970Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6963185Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6963413Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6963516Z ok (6.520s) 2023-01-11T21:44:22.6963536Z 2023-01-11T21:44:22.6963801Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6963913Z Ran 1 test in 6.520s 2023-01-11T21:44:22.6963933Z 2023-01-11T21:44:22.6964023Z OK 2023-01-11T21:44:22.6964044Z 2023-01-11T21:44:22.6964167Z Generating XML reports... 2023-01-11T21:44:22.6964616Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212855.xml 2023-01-11T21:44:22.6964966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6965141Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6965518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6965706Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6965725Z 2023-01-11T21:44:22.6965832Z Running tests... 2023-01-11T21:44:22.6966095Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6966404Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6966761Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6966981Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14945 2023-01-11T21:44:22.6967176Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14946 2023-01-11T21:44:22.6967545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6967723Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6968095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6968279Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6968637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6968810Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6969175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6969341Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6969584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6969876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6970433Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6970848Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6971075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6971305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6971536Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6971768Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6971980Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6972213Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6972314Z ok (6.502s) 2023-01-11T21:44:22.6972334Z 2023-01-11T21:44:22.6972601Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6972712Z Ran 1 test in 6.502s 2023-01-11T21:44:22.6972732Z 2023-01-11T21:44:22.6972826Z OK 2023-01-11T21:44:22.6972845Z 2023-01-11T21:44:22.6972967Z Generating XML reports... 2023-01-11T21:44:22.6973416Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212904.xml 2023-01-11T21:44:22.6973824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6973982Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6974360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6974551Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6974571Z 2023-01-11T21:44:22.6974678Z Running tests... 2023-01-11T21:44:22.6974943Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6975255Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6975620Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6975929Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15090 2023-01-11T21:44:22.6976123Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15091 2023-01-11T21:44:22.6976495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6976673Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6977048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6977239Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6977598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6977769Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6978135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6978325Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6978548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6978789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6979245Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6979650Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6979876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6980100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6980336Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6980567Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6980797Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6981005Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6981109Z ok (6.412s) 2023-01-11T21:44:22.6981132Z 2023-01-11T21:44:22.6981400Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6981512Z Ran 1 test in 6.412s 2023-01-11T21:44:22.6981531Z 2023-01-11T21:44:22.6981627Z OK 2023-01-11T21:44:22.6981646Z 2023-01-11T21:44:22.6981769Z Generating XML reports... 2023-01-11T21:44:22.6982218Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212913.xml 2023-01-11T21:44:22.6982588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6982747Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6983124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6983312Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6983332Z 2023-01-11T21:44:22.6983443Z Running tests... 2023-01-11T21:44:22.6983707Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6984016Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6984381Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6984599Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15235 2023-01-11T21:44:22.6984868Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15236 2023-01-11T21:44:22.6985220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6985393Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6985768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6985955Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6986309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6986481Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6986850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6987039Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6987261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6987501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6987896Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6988332Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6988562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6988789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6989021Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6989256Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6989479Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6989690Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6989792Z ok (6.395s) 2023-01-11T21:44:22.6989812Z 2023-01-11T21:44:22.6990084Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6990195Z Ran 1 test in 6.396s 2023-01-11T21:44:22.6990214Z 2023-01-11T21:44:22.6990306Z OK 2023-01-11T21:44:22.6990325Z 2023-01-11T21:44:22.6990449Z Generating XML reports... 2023-01-11T21:44:22.6990896Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212922.xml 2023-01-11T21:44:22.6991259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6991439Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6991797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6991984Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6992003Z 2023-01-11T21:44:22.6992112Z Running tests... 2023-01-11T21:44:22.6992375Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6992686Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.6993049Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.6993265Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15380 2023-01-11T21:44:22.6993477Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15381 2023-01-11T21:44:22.6993879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6994052Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6994424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6994615Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6994976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.6995147Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.6995516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.6995699Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.6995944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.6996164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.6996558Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6996993Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.6997226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.6997451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.6997685Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6997918Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6998146Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6998376Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.6998460Z ok (6.541s) 2023-01-11T21:44:22.6998480Z 2023-01-11T21:44:22.6998751Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.6998863Z Ran 1 test in 6.541s 2023-01-11T21:44:22.6998882Z 2023-01-11T21:44:22.6998978Z OK 2023-01-11T21:44:22.6998997Z 2023-01-11T21:44:22.6999124Z Generating XML reports... 2023-01-11T21:44:22.6999568Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212931.xml 2023-01-11T21:44:22.6999932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7000109Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7000467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7000656Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7000675Z 2023-01-11T21:44:22.7000782Z Running tests... 2023-01-11T21:44:22.7001042Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7001353Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7001719Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7001936Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15525 2023-01-11T21:44:22.7002149Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15526 2023-01-11T21:44:22.7002513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7002737Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7003116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7003302Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7003663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7003834Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7004201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7004388Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7004630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7004856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7005250Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7005640Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7005910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7006140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7006375Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.7006605Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.7006828Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.7007060Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.7007145Z ok (14.220s) 2023-01-11T21:44:22.7007165Z 2023-01-11T21:44:22.7007435Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7007548Z Ran 1 test in 14.220s 2023-01-11T21:44:22.7007567Z 2023-01-11T21:44:22.7007659Z OK 2023-01-11T21:44:22.7007678Z 2023-01-11T21:44:22.7007802Z Generating XML reports... 2023-01-11T21:44:22.7008250Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212943.xml 2023-01-11T21:44:22.7008614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7008787Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7009163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7009337Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7009356Z 2023-01-11T21:44:22.7009464Z Running tests... 2023-01-11T21:44:22.7009725Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7010033Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7010653Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7010884Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15670 2023-01-11T21:44:22.7011099Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15671 2023-01-11T21:44:22.7011471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7011707Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7012087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7012275Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7012634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7012812Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7013182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7013366Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7013606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7013846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7014229Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7014620Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7014846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7015127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7015368Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.7015598Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.7015822Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.7016048Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.7016135Z ok (6.421s) 2023-01-11T21:44:22.7016175Z 2023-01-11T21:44:22.7016428Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7016541Z Ran 1 test in 6.422s 2023-01-11T21:44:22.7016561Z 2023-01-11T21:44:22.7016653Z OK 2023-01-11T21:44:22.7016672Z 2023-01-11T21:44:22.7016796Z Generating XML reports... 2023-01-11T21:44:22.7017246Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212959.xml 2023-01-11T21:44:22.7017610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7017786Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7018162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7018333Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7018373Z 2023-01-11T21:44:22.7018464Z Running tests... 2023-01-11T21:44:22.7018725Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7019033Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7019396Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7019614Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15815 2023-01-11T21:44:22.7019827Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15816 2023-01-11T21:44:22.7020190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7020362Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7020775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7022526Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7022900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7023072Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7023449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7023634Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7023874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7024115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7024507Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7024885Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7025109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7025333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7025619Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.7025856Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.7026081Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.7026311Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.7026415Z ok (6.434s) 2023-01-11T21:44:22.7026437Z 2023-01-11T21:44:22.7026707Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7026802Z Ran 1 test in 6.434s 2023-01-11T21:44:22.7026821Z 2023-01-11T21:44:22.7026914Z OK 2023-01-11T21:44:22.7026933Z 2023-01-11T21:44:22.7027058Z Generating XML reports... 2023-01-11T21:44:22.7027505Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213008.xml 2023-01-11T21:44:22.7027871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7028045Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7028417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7028606Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7028625Z 2023-01-11T21:44:22.7028715Z Running tests... 2023-01-11T21:44:22.7028983Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7029291Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7029649Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7029868Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15960 2023-01-11T21:44:22.7030081Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15961 2023-01-11T21:44:22.7030443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7030615Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7030990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7031214Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7031575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7031747Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7032112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7032299Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7032541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7032781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7033177Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7033571Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7033780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7034005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7034235Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.7034510Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.7034742Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.7034971Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.7035074Z ok (6.399s) 2023-01-11T21:44:22.7035094Z 2023-01-11T21:44:22.7035366Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7035463Z Ran 1 test in 6.399s 2023-01-11T21:44:22.7035482Z 2023-01-11T21:44:22.7035575Z OK 2023-01-11T21:44:22.7035594Z 2023-01-11T21:44:22.7035717Z Generating XML reports... 2023-01-11T21:44:22.7036164Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213017.xml 2023-01-11T21:44:22.7036529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7036706Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7037078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7037265Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7037284Z 2023-01-11T21:44:22.7037393Z Running tests... 2023-01-11T21:44:22.7037638Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7037949Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7038305Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7038523Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16105 2023-01-11T21:44:22.7038738Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16106 2023-01-11T21:44:22.7039105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7039277Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7039652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7039821Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7040234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7040404Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7040770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7040956Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7041203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7041443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7041838Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7042230Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7042439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7042662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7042893Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.7043120Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.7043388Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.7043624Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.7043728Z ok (6.535s) 2023-01-11T21:44:22.7043747Z 2023-01-11T21:44:22.7044015Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7044109Z Ran 1 test in 6.535s 2023-01-11T21:44:22.7044147Z 2023-01-11T21:44:22.7044224Z OK 2023-01-11T21:44:22.7044243Z 2023-01-11T21:44:22.7044366Z Generating XML reports... 2023-01-11T21:44:22.7044812Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213026.xml 2023-01-11T21:44:22.7045175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7045345Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7045718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7045906Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7045925Z 2023-01-11T21:44:22.7046034Z Running tests... 2023-01-11T21:44:22.7046280Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7046589Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7046897Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7047114Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16250 2023-01-11T21:44:22.7047327Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16251 2023-01-11T21:44:22.7047696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7047870Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7048247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7048435Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7048777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7049006Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7049375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7049563Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7049804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7050048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7050620Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7051023Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7051230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7051460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7051691Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.7051924Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.7052149Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.7052451Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.7052565Z ok (6.502s) 2023-01-11T21:44:22.7052585Z 2023-01-11T21:44:22.7052859Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7052971Z Ran 1 test in 6.502s 2023-01-11T21:44:22.7052990Z 2023-01-11T21:44:22.7053064Z OK 2023-01-11T21:44:22.7053082Z 2023-01-11T21:44:22.7053206Z Generating XML reports... 2023-01-11T21:44:22.7053657Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213035.xml 2023-01-11T21:44:22.7054026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7054199Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7054573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7054764Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7054783Z 2023-01-11T21:44:22.7054892Z Running tests... 2023-01-11T21:44:22.7055136Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7055449Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7055753Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7055975Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16395 2023-01-11T21:44:22.7056188Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16396 2023-01-11T21:44:22.7056553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7056726Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7057102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7057289Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7057631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7057804Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7058170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7058427Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7058670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7059070Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7059313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7059705Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7059931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7060137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7060371Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.7060600Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.7060823Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.7061053Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.7061154Z ok (6.442s) 2023-01-11T21:44:22.7061173Z 2023-01-11T21:44:22.7061496Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7061613Z Ran 1 test in 6.442s 2023-01-11T21:44:22.7061632Z 2023-01-11T21:44:22.7061705Z OK 2023-01-11T21:44:22.7061724Z 2023-01-11T21:44:22.7061846Z Generating XML reports... 2023-01-11T21:44:22.7062297Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213044.xml 2023-01-11T21:44:22.7062664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7062838Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7063208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7063395Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7063414Z 2023-01-11T21:44:22.7063523Z Running tests... 2023-01-11T21:44:22.7063792Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7064085Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7064345Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7065091Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77325 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.641s) 2023-01-11T21:44:22.7065115Z 2023-01-11T21:44:22.7065377Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7065489Z Ran 1 test in 1.641s 2023-01-11T21:44:22.7065508Z 2023-01-11T21:44:22.7065614Z OK (skipped=1) 2023-01-11T21:44:22.7065633Z 2023-01-11T21:44:22.7065757Z Generating XML reports... 2023-01-11T21:44:22.7066204Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213053.xml 2023-01-11T21:44:22.7066572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7066726Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7067101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7067357Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7067377Z 2023-01-11T21:44:22.7067486Z Running tests... 2023-01-11T21:44:22.7067751Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7068061Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7068315Z test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7068535Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16574 2023-01-11T21:44:22.7068748Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16575 2023-01-11T21:44:22.7069101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7069274Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7069652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7069841Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7070201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7070374Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7070794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7070986Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7071209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7071450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7071855Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7072245Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7072471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7072698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7072800Z ok (6.642s) 2023-01-11T21:44:22.7072819Z 2023-01-11T21:44:22.7073084Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7073196Z Ran 1 test in 6.642s 2023-01-11T21:44:22.7073215Z 2023-01-11T21:44:22.7073288Z OK 2023-01-11T21:44:22.7073307Z 2023-01-11T21:44:22.7073430Z Generating XML reports... 2023-01-11T21:44:22.7073923Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213058.xml 2023-01-11T21:44:22.7074294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7074468Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7074840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7075029Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7075052Z 2023-01-11T21:44:22.7075161Z Running tests... 2023-01-11T21:44:22.7075404Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7075715Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7075981Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7076197Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16685 2023-01-11T21:44:22.7076465Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16686 2023-01-11T21:44:22.7076835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7077008Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7077382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7077570Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7077911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7078084Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7078449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7078634Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7078877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7079118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7079512Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7079946Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7080177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7080381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7080615Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.7080847Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.7081246Z /opt/conda/lib/python3.10/tempfile.py:860: ResourceWarning: Implicitly cleaning up 2023-01-11T21:44:22.7081410Z _warnings.warn(warn_message, ResourceWarning) 2023-01-11T21:44:22.7081801Z /opt/conda/lib/python3.10/tempfile.py:860: ResourceWarning: Implicitly cleaning up 2023-01-11T21:44:22.7081966Z _warnings.warn(warn_message, ResourceWarning) 2023-01-11T21:44:22.7082065Z ok (6.139s) 2023-01-11T21:44:22.7082085Z 2023-01-11T21:44:22.7082332Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7082444Z Ran 1 test in 6.139s 2023-01-11T21:44:22.7082464Z 2023-01-11T21:44:22.7082556Z OK 2023-01-11T21:44:22.7082575Z 2023-01-11T21:44:22.7082698Z Generating XML reports... 2023-01-11T21:44:22.7083141Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213107.xml 2023-01-11T21:44:22.7083508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7083681Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7084053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7084245Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7084264Z 2023-01-11T21:44:22.7084355Z Running tests... 2023-01-11T21:44:22.7084624Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7084933Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7085192Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7085408Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16800 2023-01-11T21:44:22.7085680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16801 2023-01-11T21:44:22.7086046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7086221Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7086578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7086768Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7087124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7087295Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7087663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7087848Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7088090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7088334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7088775Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7089154Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7089380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7089604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7089836Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.7090071Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.7090172Z ok (4.236s) 2023-01-11T21:44:22.7090192Z 2023-01-11T21:44:22.7090634Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7090753Z Ran 1 test in 4.236s 2023-01-11T21:44:22.7090775Z 2023-01-11T21:44:22.7090849Z OK 2023-01-11T21:44:22.7090868Z 2023-01-11T21:44:22.7090994Z Generating XML reports... 2023-01-11T21:44:22.7091450Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213115.xml 2023-01-11T21:44:22.7091817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7091990Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7092363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7092556Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7092576Z 2023-01-11T21:44:22.7092683Z Running tests... 2023-01-11T21:44:22.7092948Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7093237Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7093501Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7093715Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16943 2023-01-11T21:44:22.7093930Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16944 2023-01-11T21:44:22.7094297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7094470Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7094985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7095172Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7095513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7095686Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7096058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7096242Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7096482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7096718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7097118Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7097511Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7097734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7097938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7098233Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.7098473Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.7098576Z ok (6.266s) 2023-01-11T21:44:22.7098595Z 2023-01-11T21:44:22.7098867Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7098979Z Ran 1 test in 6.266s 2023-01-11T21:44:22.7098998Z 2023-01-11T21:44:22.7099094Z OK 2023-01-11T21:44:22.7099113Z 2023-01-11T21:44:22.7099235Z Generating XML reports... 2023-01-11T21:44:22.7099661Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213122.xml 2023-01-11T21:44:22.7100030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7100203Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7100584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7100772Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7100792Z 2023-01-11T21:44:22.7100900Z Running tests... 2023-01-11T21:44:22.7101165Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7101473Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7101757Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7101954Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17058 2023-01-11T21:44:22.7102169Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17059 2023-01-11T21:44:22.7102539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7102712Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7103082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7103272Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7103630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7103854Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7104204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7104392Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7104632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7104875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7105270Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7105661Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7105891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7106117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7106352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T21:44:22.7106570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T21:44:22.7106963Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.7107398Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.7107639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2023-01-11T21:44:22.7107873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2023-01-11T21:44:22.7108259Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2023-01-11T21:44:22.7108653Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2023-01-11T21:44:22.7108756Z ok (5.753s) 2023-01-11T21:44:22.7108776Z 2023-01-11T21:44:22.7109043Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7109137Z Ran 1 test in 5.753s 2023-01-11T21:44:22.7109178Z 2023-01-11T21:44:22.7109250Z OK 2023-01-11T21:44:22.7109273Z 2023-01-11T21:44:22.7109395Z Generating XML reports... 2023-01-11T21:44:22.7109842Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213131.xml 2023-01-11T21:44:22.7110208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7110381Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7110756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7110949Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7110968Z 2023-01-11T21:44:22.7111076Z Running tests... 2023-01-11T21:44:22.7111320Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7111628Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7111907Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7112122Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17181 2023-01-11T21:44:22.7112336Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17182 2023-01-11T21:44:22.7112702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7112932Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7113309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7113477Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7113834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7114008Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7114380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7114564Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7114805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7115044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7115442Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7115830Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7116036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7116302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7116545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T21:44:22.7116784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T21:44:22.7117177Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.7117574Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.7117810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2023-01-11T21:44:22.7118044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2023-01-11T21:44:22.7118429Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2023-01-11T21:44:22.7118797Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2023-01-11T21:44:22.7118901Z ok (15.770s) 2023-01-11T21:44:22.7118921Z 2023-01-11T21:44:22.7119186Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7119298Z Ran 1 test in 15.770s 2023-01-11T21:44:22.7119317Z 2023-01-11T21:44:22.7119407Z OK 2023-01-11T21:44:22.7119426Z 2023-01-11T21:44:22.7119549Z Generating XML reports... 2023-01-11T21:44:22.7119995Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213139.xml 2023-01-11T21:44:22.7120362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7120536Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7120896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7121085Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7121105Z 2023-01-11T21:44:22.7121211Z Running tests... 2023-01-11T21:44:22.7121472Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7121782Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7122139Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7122356Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17304 2023-01-11T21:44:22.7122571Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17305 2023-01-11T21:44:22.7122941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7123098Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7123472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7123658Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7124019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7124192Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7124564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7124749Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7124987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7125254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7125657Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7126046Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7126273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7126498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7126603Z ok (6.490s) 2023-01-11T21:44:22.7126623Z 2023-01-11T21:44:22.7126891Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7127004Z Ran 1 test in 6.490s 2023-01-11T21:44:22.7127023Z 2023-01-11T21:44:22.7127118Z OK 2023-01-11T21:44:22.7127136Z 2023-01-11T21:44:22.7127241Z Generating XML reports... 2023-01-11T21:44:22.7127690Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213158.xml 2023-01-11T21:44:22.7128058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7128231Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7128605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7128799Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7128818Z 2023-01-11T21:44:22.7128926Z Running tests... 2023-01-11T21:44:22.7129191Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7129484Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7129769Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7129986Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17419 2023-01-11T21:44:22.7130203Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17420 2023-01-11T21:44:22.7130745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7130921Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7131297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7131586Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7131949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7132100Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7132471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7132656Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7132898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7133137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7133535Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7133932Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7134156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7134380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7134464Z ok (6.514s) 2023-01-11T21:44:22.7134546Z 2023-01-11T21:44:22.7134825Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7134939Z Ran 1 test in 6.514s 2023-01-11T21:44:22.7134958Z 2023-01-11T21:44:22.7135051Z OK 2023-01-11T21:44:22.7135070Z 2023-01-11T21:44:22.7135193Z Generating XML reports... 2023-01-11T21:44:22.7135640Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213207.xml 2023-01-11T21:44:22.7136011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7136185Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7136541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7136730Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7136749Z 2023-01-11T21:44:22.7136859Z Running tests... 2023-01-11T21:44:22.7137124Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7137434Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7137688Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7137904Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17534 2023-01-11T21:44:22.7138121Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17535 2023-01-11T21:44:22.7138467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7138641Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7139017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7139208Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7139568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7139736Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7140104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7140291Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7140591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7140810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7141208Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7141600Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7141825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7142048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7142150Z ok (6.131s) 2023-01-11T21:44:22.7142170Z 2023-01-11T21:44:22.7142436Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7142552Z Ran 1 test in 6.131s 2023-01-11T21:44:22.7142571Z 2023-01-11T21:44:22.7142663Z OK 2023-01-11T21:44:22.7142682Z 2023-01-11T21:44:22.7142787Z Generating XML reports... 2023-01-11T21:44:22.7143230Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213216.xml 2023-01-11T21:44:22.7143594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7143814Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7144197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7144388Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7144407Z 2023-01-11T21:44:22.7144516Z Running tests... 2023-01-11T21:44:22.7144780Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7145076Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7145336Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7145551Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17645 2023-01-11T21:44:22.7145763Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17646 2023-01-11T21:44:22.7146128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7146299Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7146674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7146862Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7147221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7147378Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7147742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7147927Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7148168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7148409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7148804Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7149195Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7149419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7149711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7150464Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-01-11T21:44:22.7151245Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-01-11T21:44:22.7151484Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.7151699Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.7151842Z ok (6.210s) 2023-01-11T21:44:22.7151863Z 2023-01-11T21:44:22.7152142Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7152255Z Ran 1 test in 6.210s 2023-01-11T21:44:22.7152275Z 2023-01-11T21:44:22.7152370Z OK 2023-01-11T21:44:22.7152389Z 2023-01-11T21:44:22.7152513Z Generating XML reports... 2023-01-11T21:44:22.7152961Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213224.xml 2023-01-11T21:44:22.7153333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7153508Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7153867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7154056Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7154078Z 2023-01-11T21:44:22.7154186Z Running tests... 2023-01-11T21:44:22.7154454Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7154767Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7155043Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7155784Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78338 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.637s) 2023-01-11T21:44:22.7155808Z 2023-01-11T21:44:22.7156071Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7156184Z Ran 1 test in 1.638s 2023-01-11T21:44:22.7156203Z 2023-01-11T21:44:22.7156292Z OK (skipped=1) 2023-01-11T21:44:22.7156332Z 2023-01-11T21:44:22.7156438Z Generating XML reports... 2023-01-11T21:44:22.7156884Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213233.xml 2023-01-11T21:44:22.7157245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7157418Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7157857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7158047Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7158066Z 2023-01-11T21:44:22.7158174Z Running tests... 2023-01-11T21:44:22.7158438Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7158729Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7159004Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7159745Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77342 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.583s) 2023-01-11T21:44:22.7159768Z 2023-01-11T21:44:22.7160031Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7160145Z Ran 1 test in 1.583s 2023-01-11T21:44:22.7160165Z 2023-01-11T21:44:22.7160272Z OK (skipped=1) 2023-01-11T21:44:22.7160290Z 2023-01-11T21:44:22.7160417Z Generating XML reports... 2023-01-11T21:44:22.7160867Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213237.xml 2023-01-11T21:44:22.7161278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7161461Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7161821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7162011Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7162029Z 2023-01-11T21:44:22.7162138Z Running tests... 2023-01-11T21:44:22.7162406Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7162716Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7162987Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7163202Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17828 2023-01-11T21:44:22.7163418Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17829 2023-01-11T21:44:22.7163763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7163935Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7164311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7164499Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7164863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7165032Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7165400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7165587Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7165830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7166052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7166448Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7166842Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7167123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7167346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7167684Z STAGE:2023-01-11 21:32:47 17828:17828 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7168007Z STAGE:2023-01-11 21:32:47 17829:17829 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7168242Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.7168457Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.7168999Z STAGE:2023-01-11 21:32:48 17829:17829 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:32:48 17828:17828 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7169023Z 2023-01-11T21:44:22.7169370Z STAGE:2023-01-11 21:32:48 17829:17829 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7169711Z STAGE:2023-01-11 21:32:48 17828:17828 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7170783Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-01-11T21:44:22.7171567Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-01-11T21:44:22.7172108Z STAGE:2023-01-11 21:32:48 17829:17829 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:32:48 17828:17828 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7172128Z 2023-01-11T21:44:22.7172457Z STAGE:2023-01-11 21:32:48 17828:17828 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7172779Z STAGE:2023-01-11 21:32:48 17829:17829 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7173118Z STAGE:2023-01-11 21:32:48 17828:17828 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7173463Z STAGE:2023-01-11 21:32:48 17829:17829 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7173567Z ok (6.637s) 2023-01-11T21:44:22.7173586Z 2023-01-11T21:44:22.7173892Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7174004Z Ran 1 test in 6.638s 2023-01-11T21:44:22.7174023Z 2023-01-11T21:44:22.7174097Z OK 2023-01-11T21:44:22.7174134Z 2023-01-11T21:44:22.7174243Z Generating XML reports... 2023-01-11T21:44:22.7174691Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213241.xml 2023-01-11T21:44:22.7175060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7175234Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7175610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7175875Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7175894Z 2023-01-11T21:44:22.7176003Z Running tests... 2023-01-11T21:44:22.7176270Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7176561Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7176828Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7177046Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17947 2023-01-11T21:44:22.7177261Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17948 2023-01-11T21:44:22.7177625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7177799Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7178176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7178365Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7178727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7178880Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7179301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7179492Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7179737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7179977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7180379Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7180769Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7180996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7181203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7181305Z ok (5.717s) 2023-01-11T21:44:22.7181324Z 2023-01-11T21:44:22.7181590Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7181701Z Ran 1 test in 5.717s 2023-01-11T21:44:22.7181720Z 2023-01-11T21:44:22.7181813Z OK 2023-01-11T21:44:22.7181831Z 2023-01-11T21:44:22.7181954Z Generating XML reports... 2023-01-11T21:44:22.7182398Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213251.xml 2023-01-11T21:44:22.7182766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7182936Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7183292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7183482Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7183501Z 2023-01-11T21:44:22.7183611Z Running tests... 2023-01-11T21:44:22.7183876Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7184185Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7184456Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7185192Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78595 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.648s) 2023-01-11T21:44:22.7185262Z 2023-01-11T21:44:22.7185536Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7185647Z Ran 1 test in 1.648s 2023-01-11T21:44:22.7185666Z 2023-01-11T21:44:22.7185758Z OK (skipped=1) 2023-01-11T21:44:22.7185795Z 2023-01-11T21:44:22.7185900Z Generating XML reports... 2023-01-11T21:44:22.7186349Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213259.xml 2023-01-11T21:44:22.7186713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7186885Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7187264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7187449Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7187468Z 2023-01-11T21:44:22.7187576Z Running tests... 2023-01-11T21:44:22.7187837Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7188176Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7188460Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7188677Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18092 2023-01-11T21:44:22.7188894Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18093 2023-01-11T21:44:22.7189262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7189442Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7189818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7190006Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7190353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7190526Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7190901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7191089Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7191332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7191579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7191974Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7192364Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7192593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7192800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7193698Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2023-01-11T21:44:22.7193881Z warnings.warn( 2023-01-11T21:44:22.7194783Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2023-01-11T21:44:22.7194898Z warnings.warn( 2023-01-11T21:44:22.7195134Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.7195359Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.7195461Z ok (6.207s) 2023-01-11T21:44:22.7195480Z 2023-01-11T21:44:22.7195747Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7195859Z Ran 1 test in 6.207s 2023-01-11T21:44:22.7195881Z 2023-01-11T21:44:22.7195955Z OK 2023-01-11T21:44:22.7195989Z 2023-01-11T21:44:22.7196094Z Generating XML reports... 2023-01-11T21:44:22.7196545Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213303.xml 2023-01-11T21:44:22.7196912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7197086Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7197513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7197708Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7197728Z 2023-01-11T21:44:22.7197835Z Running tests... 2023-01-11T21:44:22.7198099Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7198391Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7198667Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7199407Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.642s) 2023-01-11T21:44:22.7199431Z 2023-01-11T21:44:22.7199690Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7199802Z Ran 1 test in 1.642s 2023-01-11T21:44:22.7199821Z 2023-01-11T21:44:22.7199928Z OK (skipped=1) 2023-01-11T21:44:22.7199947Z 2023-01-11T21:44:22.7200070Z Generating XML reports... 2023-01-11T21:44:22.7200513Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213312.xml 2023-01-11T21:44:22.7200886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7201064Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7201426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7201616Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7201635Z 2023-01-11T21:44:22.7201746Z Running tests... 2023-01-11T21:44:22.7202011Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7202320Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7202589Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7202806Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18241 2023-01-11T21:44:22.7203078Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18242 2023-01-11T21:44:22.7203427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7203599Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7203971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7204162Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7204527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7204697Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7205064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7205251Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7205492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7205713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7206105Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7206542Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7206775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7207001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7207337Z STAGE:2023-01-11 21:33:22 18241:18241 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7207714Z STAGE:2023-01-11 21:33:22 18242:18242 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7207950Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.7208184Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T21:44:22.7208714Z STAGE:2023-01-11 21:33:22 18242:18242 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:33:22 18241:18241 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7208752Z 2023-01-11T21:44:22.7209080Z STAGE:2023-01-11 21:33:22 18241:18241 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7209420Z STAGE:2023-01-11 21:33:22 18242:18242 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7209741Z STAGE:2023-01-11 21:33:22 18241:18241 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7210074Z STAGE:2023-01-11 21:33:22 18241:18241 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7210603Z STAGE:2023-01-11 21:33:22 18241:18241 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7210713Z ok (6.956s) 2023-01-11T21:44:22.7210733Z 2023-01-11T21:44:22.7211006Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7211119Z Ran 1 test in 6.957s 2023-01-11T21:44:22.7211138Z 2023-01-11T21:44:22.7211212Z OK 2023-01-11T21:44:22.7211235Z 2023-01-11T21:44:22.7211360Z Generating XML reports... 2023-01-11T21:44:22.7211810Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213316.xml 2023-01-11T21:44:22.7212176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7212352Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7212819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7213007Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7213027Z 2023-01-11T21:44:22.7213135Z Running tests... 2023-01-11T21:44:22.7213399Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7213693Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7213955Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7214175Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18360 2023-01-11T21:44:22.7214390Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18361 2023-01-11T21:44:22.7214756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7214932Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7215305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7215494Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7215833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7216066Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7216449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7216635Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7216877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7217119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7217518Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7217909Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7218135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7218344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7218449Z ok (5.746s) 2023-01-11T21:44:22.7218468Z 2023-01-11T21:44:22.7218732Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7218845Z Ran 1 test in 5.746s 2023-01-11T21:44:22.7218864Z 2023-01-11T21:44:22.7218960Z OK 2023-01-11T21:44:22.7218978Z 2023-01-11T21:44:22.7219101Z Generating XML reports... 2023-01-11T21:44:22.7219546Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213326.xml 2023-01-11T21:44:22.7219913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7220067Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7220438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7220627Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7220646Z 2023-01-11T21:44:22.7220754Z Running tests... 2023-01-11T21:44:22.7221016Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7221330Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7221599Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7221874Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18471 2023-01-11T21:44:22.7222089Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18472 2023-01-11T21:44:22.7222439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7222611Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7222986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7223174Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7223533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7223706Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7224074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7224258Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7224482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7224879Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7225162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7225554Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7225785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7226009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7226115Z ok (5.728s) 2023-01-11T21:44:22.7226135Z 2023-01-11T21:44:22.7226404Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7226516Z Ran 1 test in 5.728s 2023-01-11T21:44:22.7226536Z 2023-01-11T21:44:22.7226609Z OK 2023-01-11T21:44:22.7226627Z 2023-01-11T21:44:22.7226750Z Generating XML reports... 2023-01-11T21:44:22.7227196Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213334.xml 2023-01-11T21:44:22.7227565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7227740Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7228113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7228302Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7228321Z 2023-01-11T21:44:22.7228434Z Running tests... 2023-01-11T21:44:22.7228702Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7228995Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7229268Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7230015Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78684 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.592s) 2023-01-11T21:44:22.7230036Z 2023-01-11T21:44:22.7230299Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7230411Z Ran 1 test in 1.592s 2023-01-11T21:44:22.7230430Z 2023-01-11T21:44:22.7230536Z OK (skipped=1) 2023-01-11T21:44:22.7230555Z 2023-01-11T21:44:22.7230731Z Generating XML reports... 2023-01-11T21:44:22.7231180Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213342.xml 2023-01-11T21:44:22.7231541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7231697Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7232071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7232259Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7232279Z 2023-01-11T21:44:22.7232385Z Running tests... 2023-01-11T21:44:22.7232649Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7232959Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7233214Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7233951Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.591s) 2023-01-11T21:44:22.7233971Z 2023-01-11T21:44:22.7234330Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7234450Z Ran 1 test in 1.591s 2023-01-11T21:44:22.7234469Z 2023-01-11T21:44:22.7234558Z OK (skipped=1) 2023-01-11T21:44:22.7234576Z 2023-01-11T21:44:22.7234700Z Generating XML reports... 2023-01-11T21:44:22.7235149Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213346.xml 2023-01-11T21:44:22.7235518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7235694Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7236071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7236260Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7236279Z 2023-01-11T21:44:22.7236387Z Running tests... 2023-01-11T21:44:22.7236631Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7236939Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7237218Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7237947Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.638s) 2023-01-11T21:44:22.7237970Z 2023-01-11T21:44:22.7238232Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7238341Z Ran 1 test in 1.639s 2023-01-11T21:44:22.7238360Z 2023-01-11T21:44:22.7238466Z OK (skipped=1) 2023-01-11T21:44:22.7238485Z 2023-01-11T21:44:22.7238609Z Generating XML reports... 2023-01-11T21:44:22.7239052Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213350.xml 2023-01-11T21:44:22.7239420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7239575Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7239951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7240194Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7240213Z 2023-01-11T21:44:22.7240321Z Running tests... 2023-01-11T21:44:22.7240586Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7240896Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7241192Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7241413Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18684 2023-01-11T21:44:22.7241630Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18685 2023-01-11T21:44:22.7241977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7242154Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7242533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7242719Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7243077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7243248Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7243660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7243869Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7244079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7244319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7244719Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7245107Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7245332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7245558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7245664Z ok (6.261s) 2023-01-11T21:44:22.7245684Z 2023-01-11T21:44:22.7245951Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7246065Z Ran 1 test in 6.261s 2023-01-11T21:44:22.7246084Z 2023-01-11T21:44:22.7246158Z OK 2023-01-11T21:44:22.7246177Z 2023-01-11T21:44:22.7246299Z Generating XML reports... 2023-01-11T21:44:22.7246743Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213354.xml 2023-01-11T21:44:22.7247114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7247287Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7247659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7247848Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7247870Z 2023-01-11T21:44:22.7247979Z Running tests... 2023-01-11T21:44:22.7248241Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7248531Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7248793Z test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7249005Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18799 2023-01-11T21:44:22.7249276Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18800 2023-01-11T21:44:22.7249644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7249817Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7250192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7250549Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7250905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7251077Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7251443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7251631Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7251876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7252115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7252512Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7252976Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7253211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7253415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7253793Z /opt/conda/lib/python3.10/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op 2023-01-11T21:44:22.7254050Z warnings.warn("Initializing zero-element tensors is a no-op") 2023-01-11T21:44:22.7254419Z /opt/conda/lib/python3.10/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op 2023-01-11T21:44:22.7254668Z warnings.warn("Initializing zero-element tensors is a no-op") 2023-01-11T21:44:22.7254770Z ok (5.879s) 2023-01-11T21:44:22.7254790Z 2023-01-11T21:44:22.7255056Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7255171Z Ran 1 test in 5.879s 2023-01-11T21:44:22.7255190Z 2023-01-11T21:44:22.7255264Z OK 2023-01-11T21:44:22.7255301Z 2023-01-11T21:44:22.7255407Z Generating XML reports... 2023-01-11T21:44:22.7255849Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213403.xml 2023-01-11T21:44:22.7256212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7256387Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7256760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7256949Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7256968Z 2023-01-11T21:44:22.7257075Z Running tests... 2023-01-11T21:44:22.7257338Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7257635Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7257893Z test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7258108Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18910 2023-01-11T21:44:22.7258321Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18911 2023-01-11T21:44:22.7258686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7258945Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7259323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7259514Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7259856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7260031Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7260401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7260583Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7260824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7261067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7261463Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7261855Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7262127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7262338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7262575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T21:44:22.7262815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T21:44:22.7263208Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.7263601Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.7263703Z ok (4.387s) 2023-01-11T21:44:22.7263723Z 2023-01-11T21:44:22.7263983Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7264095Z Ran 1 test in 4.387s 2023-01-11T21:44:22.7264114Z 2023-01-11T21:44:22.7264207Z OK 2023-01-11T21:44:22.7264226Z 2023-01-11T21:44:22.7264331Z Generating XML reports... 2023-01-11T21:44:22.7264777Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213412.xml 2023-01-11T21:44:22.7265142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7265315Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7265693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7265884Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7265903Z 2023-01-11T21:44:22.7266012Z Running tests... 2023-01-11T21:44:22.7266276Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7266568Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7266820Z test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7267036Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19025 2023-01-11T21:44:22.7267250Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19026 2023-01-11T21:44:22.7267615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7267842Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7268219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7268408Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7268769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7268927Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7269296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7269481Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7269722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7269962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7270361Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7270751Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7270974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7271248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7271471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T21:44:22.7271710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T21:44:22.7272101Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.7272493Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.7272596Z ok (4.210s) 2023-01-11T21:44:22.7272616Z 2023-01-11T21:44:22.7272882Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7272994Z Ran 1 test in 4.211s 2023-01-11T21:44:22.7273014Z 2023-01-11T21:44:22.7273106Z OK 2023-01-11T21:44:22.7273124Z 2023-01-11T21:44:22.7273232Z Generating XML reports... 2023-01-11T21:44:22.7273717Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213419.xml 2023-01-11T21:44:22.7274087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7274262Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7274637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7274829Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7274848Z 2023-01-11T21:44:22.7274953Z Running tests... 2023-01-11T21:44:22.7275214Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7275522Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7275777Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7276516Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78767 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.621s) 2023-01-11T21:44:22.7276536Z 2023-01-11T21:44:22.7276801Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7276974Z Ran 1 test in 1.621s 2023-01-11T21:44:22.7276993Z 2023-01-11T21:44:22.7277100Z OK (skipped=1) 2023-01-11T21:44:22.7277119Z 2023-01-11T21:44:22.7277245Z Generating XML reports... 2023-01-11T21:44:22.7277689Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213426.xml 2023-01-11T21:44:22.7278057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7278234Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7278616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7278787Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7278806Z 2023-01-11T21:44:22.7278914Z Running tests... 2023-01-11T21:44:22.7279178Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7279493Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7279766Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7280553Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78748 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.667s) 2023-01-11T21:44:22.7280574Z 2023-01-11T21:44:22.7280844Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7280957Z Ran 1 test in 1.667s 2023-01-11T21:44:22.7280976Z 2023-01-11T21:44:22.7281085Z OK (skipped=1) 2023-01-11T21:44:22.7281104Z 2023-01-11T21:44:22.7281210Z Generating XML reports... 2023-01-11T21:44:22.7281654Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213430.xml 2023-01-11T21:44:22.7282021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7282196Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7282571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7282761Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7282780Z 2023-01-11T21:44:22.7282889Z Running tests... 2023-01-11T21:44:22.7283154Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7283463Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7283713Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7283931Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19208 2023-01-11T21:44:22.7284144Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19209 2023-01-11T21:44:22.7284510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7284680Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7285056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7285247Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7285604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7285755Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7286120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7286361Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7286604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7286847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7287250Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7287637Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7287860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7288089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7288174Z ok (4.157s) 2023-01-11T21:44:22.7288196Z 2023-01-11T21:44:22.7288461Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7288574Z Ran 1 test in 4.157s 2023-01-11T21:44:22.7288593Z 2023-01-11T21:44:22.7288685Z OK 2023-01-11T21:44:22.7288704Z 2023-01-11T21:44:22.7288827Z Generating XML reports... 2023-01-11T21:44:22.7289271Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213434.xml 2023-01-11T21:44:22.7289682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7289862Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7290238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7290569Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7290589Z 2023-01-11T21:44:22.7290711Z Running tests... 2023-01-11T21:44:22.7290978Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7291287Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7291524Z test_gather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7291742Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19317 2023-01-11T21:44:22.7291960Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19318 2023-01-11T21:44:22.7292323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7292475Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7292852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7293040Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7293401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7293570Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7293939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7294128Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7294373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7294610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7294988Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7295379Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7295694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7295918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7296256Z STAGE:2023-01-11 21:34:44 19318:19318 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7296578Z STAGE:2023-01-11 21:34:44 19317:19317 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7296909Z STAGE:2023-01-11 21:34:44 19318:19318 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7297256Z STAGE:2023-01-11 21:34:44 19318:19318 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7297584Z STAGE:2023-01-11 21:34:44 19317:19317 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7297907Z STAGE:2023-01-11 21:34:44 19317:19317 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7298230Z STAGE:2023-01-11 21:34:44 19318:19318 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7298548Z STAGE:2023-01-11 21:34:44 19317:19317 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7298874Z STAGE:2023-01-11 21:34:44 19317:19317 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7299276Z STAGE:2023-01-11 21:34:44 19317:19317 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7299616Z STAGE:2023-01-11 21:34:44 19318:19318 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7299959Z STAGE:2023-01-11 21:34:44 19318:19318 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7300062Z ok (4.255s) 2023-01-11T21:44:22.7300082Z 2023-01-11T21:44:22.7300327Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7300442Z Ran 1 test in 4.255s 2023-01-11T21:44:22.7300462Z 2023-01-11T21:44:22.7300555Z OK 2023-01-11T21:44:22.7300574Z 2023-01-11T21:44:22.7300695Z Generating XML reports... 2023-01-11T21:44:22.7301139Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213441.xml 2023-01-11T21:44:22.7301508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7301682Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7302054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7302242Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7302261Z 2023-01-11T21:44:22.7302351Z Running tests... 2023-01-11T21:44:22.7302613Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7302925Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7303174Z test_gather_checks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7303390Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19430 2023-01-11T21:44:22.7303606Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19431 2023-01-11T21:44:22.7303975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7304145Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7304503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7304691Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7305051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7305278Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7305646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7305832Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7306080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7306319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7306711Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7307085Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7307316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7307540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7307643Z ok (4.238s) 2023-01-11T21:44:22.7307663Z 2023-01-11T21:44:22.7307929Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7308042Z Ran 1 test in 4.238s 2023-01-11T21:44:22.7308061Z 2023-01-11T21:44:22.7308155Z OK 2023-01-11T21:44:22.7308174Z 2023-01-11T21:44:22.7308342Z Generating XML reports... 2023-01-11T21:44:22.7308781Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213447.xml 2023-01-11T21:44:22.7309150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7309323Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7309699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7309887Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7309907Z 2023-01-11T21:44:22.7310015Z Running tests... 2023-01-11T21:44:22.7310273Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7310582Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7310836Z test_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s) 2023-01-11T21:44:22.7310856Z 2023-01-11T21:44:22.7311100Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7311212Z Ran 1 test in 0.002s 2023-01-11T21:44:22.7311231Z 2023-01-11T21:44:22.7311337Z OK (skipped=1) 2023-01-11T21:44:22.7311356Z 2023-01-11T21:44:22.7311478Z Generating XML reports... 2023-01-11T21:44:22.7311920Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213454.xml 2023-01-11T21:44:22.7312290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7312460Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7312831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7313022Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7313041Z 2023-01-11T21:44:22.7313131Z Running tests... 2023-01-11T21:44:22.7313394Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7313702Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7313956Z test_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7314226Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19572 2023-01-11T21:44:22.7314440Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19573 2023-01-11T21:44:22.7314807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7314979Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7315337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7315525Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7315885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7316056Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7316422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7316610Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7316852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7317093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7317534Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7317919Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7318147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7318367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7318607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T21:44:22.7318843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T21:44:22.7319231Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.7319617Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.7319953Z STAGE:2023-01-11 21:35:01 19572:19572 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7320280Z STAGE:2023-01-11 21:35:01 19573:19573 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7320591Z STAGE:2023-01-11 21:35:01 19573:19573 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7320934Z STAGE:2023-01-11 21:35:01 19573:19573 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7321267Z STAGE:2023-01-11 21:35:01 19572:19572 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7321607Z STAGE:2023-01-11 21:35:01 19572:19572 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7321931Z STAGE:2023-01-11 21:35:01 19573:19573 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7322248Z STAGE:2023-01-11 21:35:01 19572:19572 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7322576Z STAGE:2023-01-11 21:35:01 19572:19572 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7322918Z STAGE:2023-01-11 21:35:01 19572:19572 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7323248Z STAGE:2023-01-11 21:35:01 19573:19573 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7323567Z STAGE:2023-01-11 21:35:01 19573:19573 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7323741Z ok (4.274s) 2023-01-11T21:44:22.7323761Z 2023-01-11T21:44:22.7324027Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7324139Z Ran 1 test in 4.274s 2023-01-11T21:44:22.7324159Z 2023-01-11T21:44:22.7324251Z OK 2023-01-11T21:44:22.7324269Z 2023-01-11T21:44:22.7324393Z Generating XML reports... 2023-01-11T21:44:22.7324840Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213457.xml 2023-01-11T21:44:22.7325207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7325380Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7325735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7325923Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7325946Z 2023-01-11T21:44:22.7326053Z Running tests... 2023-01-11T21:44:22.7326314Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7326623Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7326868Z test_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7327130Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19691 2023-01-11T21:44:22.7327349Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19692 2023-01-11T21:44:22.7327701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7327875Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7328251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7328444Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7328803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7328972Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7329339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7329525Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7329750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7329987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7330547Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7330957Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7331184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7331411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7331570Z skip: Skipped due to small world size. (4.222s) 2023-01-11T21:44:22.7331591Z 2023-01-11T21:44:22.7331860Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7331972Z Ran 1 test in 4.222s 2023-01-11T21:44:22.7331991Z 2023-01-11T21:44:22.7332080Z OK (skipped=1) 2023-01-11T21:44:22.7332114Z 2023-01-11T21:44:22.7332219Z Generating XML reports... 2023-01-11T21:44:22.7332671Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213504.xml 2023-01-11T21:44:22.7333034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7333288Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7333663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7333851Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7333871Z 2023-01-11T21:44:22.7333981Z Running tests... 2023-01-11T21:44:22.7334249Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7334539Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7334789Z test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7335005Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19800 2023-01-11T21:44:22.7335220Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19801 2023-01-11T21:44:22.7335592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7335764Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7336133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7336380Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7336732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7336905Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7337272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7337454Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7337700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7337941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7338338Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7338732Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7338959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7339163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7339267Z ok (4.347s) 2023-01-11T21:44:22.7339287Z 2023-01-11T21:44:22.7339553Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7339663Z Ran 1 test in 4.348s 2023-01-11T21:44:22.7339686Z 2023-01-11T21:44:22.7339778Z OK 2023-01-11T21:44:22.7339797Z 2023-01-11T21:44:22.7339917Z Generating XML reports... 2023-01-11T21:44:22.7340365Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213510.xml 2023-01-11T21:44:22.7340730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7340907Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7341264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7341451Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7341470Z 2023-01-11T21:44:22.7341577Z Running tests... 2023-01-11T21:44:22.7341841Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7342148Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7342463Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7343210Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.641s) 2023-01-11T21:44:22.7343231Z 2023-01-11T21:44:22.7343492Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7343604Z Ran 1 test in 1.641s 2023-01-11T21:44:22.7343624Z 2023-01-11T21:44:22.7343712Z OK (skipped=1) 2023-01-11T21:44:22.7343749Z 2023-01-11T21:44:22.7343855Z Generating XML reports... 2023-01-11T21:44:22.7344299Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213517.xml 2023-01-11T21:44:22.7344666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7344840Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7345216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7345405Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7345473Z 2023-01-11T21:44:22.7345587Z Running tests... 2023-01-11T21:44:22.7345854Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7346147Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7346390Z test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7346608Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19943 2023-01-11T21:44:22.7346826Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19944 2023-01-11T21:44:22.7347194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7347366Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7347740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7347929Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7348272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7348442Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7348809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7348997Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7349239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7349480Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7349876Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7350267Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7350491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7350695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7350932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T21:44:22.7351226Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T21:44:22.7351620Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.7352010Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.7352113Z ok (4.360s) 2023-01-11T21:44:22.7352133Z 2023-01-11T21:44:22.7352400Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7352511Z Ran 1 test in 4.360s 2023-01-11T21:44:22.7352531Z 2023-01-11T21:44:22.7352622Z OK 2023-01-11T21:44:22.7352641Z 2023-01-11T21:44:22.7352746Z Generating XML reports... 2023-01-11T21:44:22.7353194Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213521.xml 2023-01-11T21:44:22.7353560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7353733Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7354106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7354293Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7354313Z 2023-01-11T21:44:22.7354421Z Running tests... 2023-01-11T21:44:22.7354728Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7355028Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7355273Z test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7355488Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20058 2023-01-11T21:44:22.7355700Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20059 2023-01-11T21:44:22.7356069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7356242Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7356613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7356800Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7357161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7357311Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7357687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7357870Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7358114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7358354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7358747Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7359141Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7359367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7359590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7359674Z ok (4.171s) 2023-01-11T21:44:22.7359694Z 2023-01-11T21:44:22.7359958Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7360069Z Ran 1 test in 4.171s 2023-01-11T21:44:22.7360136Z 2023-01-11T21:44:22.7360231Z OK 2023-01-11T21:44:22.7360250Z 2023-01-11T21:44:22.7360376Z Generating XML reports... 2023-01-11T21:44:22.7360823Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213528.xml 2023-01-11T21:44:22.7361191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7361364Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7361722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7361909Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7361928Z 2023-01-11T21:44:22.7362035Z Running tests... 2023-01-11T21:44:22.7362294Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7362603Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7362845Z test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7363061Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20167 2023-01-11T21:44:22.7363275Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20168 2023-01-11T21:44:22.7363623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7363843Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7364226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7364412Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7364772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7364948Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7365312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7365495Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7365736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7365960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7366354Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7366747Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7366971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7367198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7367300Z ok (4.264s) 2023-01-11T21:44:22.7367320Z 2023-01-11T21:44:22.7367585Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7367696Z Ran 1 test in 4.264s 2023-01-11T21:44:22.7367715Z 2023-01-11T21:44:22.7367807Z OK 2023-01-11T21:44:22.7367826Z 2023-01-11T21:44:22.7367932Z Generating XML reports... 2023-01-11T21:44:22.7368379Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213535.xml 2023-01-11T21:44:22.7368751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7368924Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7369298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7369539Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7369559Z 2023-01-11T21:44:22.7369668Z Running tests... 2023-01-11T21:44:22.7369936Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7370226Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7370653Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7370878Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20276 2023-01-11T21:44:22.7371090Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20277 2023-01-11T21:44:22.7371456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7371627Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7372000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7372190Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7372550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7372703Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7373140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7373332Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7373575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7373850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7374249Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7374645Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7374870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7375075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7375313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T21:44:22.7375550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T21:44:22.7375944Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.7376331Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.7376441Z ok (4.280s) 2023-01-11T21:44:22.7376461Z 2023-01-11T21:44:22.7376729Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7376842Z Ran 1 test in 4.280s 2023-01-11T21:44:22.7376861Z 2023-01-11T21:44:22.7376954Z OK 2023-01-11T21:44:22.7376973Z 2023-01-11T21:44:22.7377078Z Generating XML reports... 2023-01-11T21:44:22.7377527Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213542.xml 2023-01-11T21:44:22.7377893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7378066Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7378443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7378631Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7378769Z 2023-01-11T21:44:22.7378883Z Running tests... 2023-01-11T21:44:22.7379149Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7379458Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7379698Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7379917Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20391 2023-01-11T21:44:22.7380132Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20392 2023-01-11T21:44:22.7380495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7380667Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7381040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7381230Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7381588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7381739Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7382112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7382343Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7382590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7382990Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7383228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7383622Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7383847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7384072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7384290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T21:44:22.7384530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T21:44:22.7384919Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.7385309Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.7385411Z ok (4.275s) 2023-01-11T21:44:22.7385434Z 2023-01-11T21:44:22.7385699Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7385810Z Ran 1 test in 4.275s 2023-01-11T21:44:22.7385830Z 2023-01-11T21:44:22.7385922Z OK 2023-01-11T21:44:22.7385940Z 2023-01-11T21:44:22.7386063Z Generating XML reports... 2023-01-11T21:44:22.7386489Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213549.xml 2023-01-11T21:44:22.7386858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7387030Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7387404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7387591Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7387610Z 2023-01-11T21:44:22.7387718Z Running tests... 2023-01-11T21:44:22.7388056Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7388364Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7388605Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7388823Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20506 2023-01-11T21:44:22.7389041Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20507 2023-01-11T21:44:22.7389409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7389582Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7389954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7390140Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7390502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7390672Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7391021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7391202Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7391487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7391733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7392127Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7392518Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7392750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7392974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7393058Z ok (6.170s) 2023-01-11T21:44:22.7393095Z 2023-01-11T21:44:22.7393342Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7393453Z Ran 1 test in 6.170s 2023-01-11T21:44:22.7393475Z 2023-01-11T21:44:22.7393567Z OK 2023-01-11T21:44:22.7393586Z 2023-01-11T21:44:22.7393710Z Generating XML reports... 2023-01-11T21:44:22.7394157Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213556.xml 2023-01-11T21:44:22.7394521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7394696Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7395072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7395241Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7395278Z 2023-01-11T21:44:22.7395368Z Running tests... 2023-01-11T21:44:22.7395631Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7395942Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7396176Z test_irecv (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7396391Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20621 2023-01-11T21:44:22.7396604Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20622 2023-01-11T21:44:22.7396967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7397173Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7397552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7397740Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7398101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7398274Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7398641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7398825Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7399064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7399307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7399682Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7400074Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7400344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7400575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7400677Z ok (4.222s) 2023-01-11T21:44:22.7400696Z 2023-01-11T21:44:22.7400961Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7401073Z Ran 1 test in 4.222s 2023-01-11T21:44:22.7401093Z 2023-01-11T21:44:22.7401185Z OK 2023-01-11T21:44:22.7401204Z 2023-01-11T21:44:22.7401327Z Generating XML reports... 2023-01-11T21:44:22.7401760Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213604.xml 2023-01-11T21:44:22.7402123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7402297Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7402670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7402858Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7402877Z 2023-01-11T21:44:22.7402987Z Running tests... 2023-01-11T21:44:22.7403251Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7403558Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7403774Z test_isend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7403992Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20730 2023-01-11T21:44:22.7404203Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20731 2023-01-11T21:44:22.7404569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7404740Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7405120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7405301Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7405664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7405836Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7406183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7406423Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7406668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7406995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7407544Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7407937Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7408164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7408391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7408480Z ok (4.284s) 2023-01-11T21:44:22.7408518Z 2023-01-11T21:44:22.7408765Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7408879Z Ran 1 test in 4.285s 2023-01-11T21:44:22.7408898Z 2023-01-11T21:44:22.7408991Z OK 2023-01-11T21:44:22.7409011Z 2023-01-11T21:44:22.7409136Z Generating XML reports... 2023-01-11T21:44:22.7409685Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213611.xml 2023-01-11T21:44:22.7410126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7410447Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7410840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7411008Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7411033Z 2023-01-11T21:44:22.7411143Z Running tests... 2023-01-11T21:44:22.7411408Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7411716Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7411980Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7412192Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20839 2023-01-11T21:44:22.7412409Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20840 2023-01-11T21:44:22.7412776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7412930Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7413301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7413491Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7413850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7414022Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7414388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7414573Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7414813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7415052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7415430Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7415912Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7416134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7416359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7416692Z STAGE:2023-01-11 21:36:22 20839:20839 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7417016Z STAGE:2023-01-11 21:36:22 20840:20840 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7417348Z STAGE:2023-01-11 21:36:22 20839:20839 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7417690Z STAGE:2023-01-11 21:36:22 20839:20839 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7418020Z STAGE:2023-01-11 21:36:22 20840:20840 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7418346Z STAGE:2023-01-11 21:36:22 20840:20840 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7418451Z ok (4.361s) 2023-01-11T21:44:22.7418470Z 2023-01-11T21:44:22.7418735Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7418846Z Ran 1 test in 4.361s 2023-01-11T21:44:22.7418865Z 2023-01-11T21:44:22.7418958Z OK 2023-01-11T21:44:22.7418977Z 2023-01-11T21:44:22.7419103Z Generating XML reports... 2023-01-11T21:44:22.7419611Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213618.xml 2023-01-11T21:44:22.7419989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7420144Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7420516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7420706Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7420725Z 2023-01-11T21:44:22.7420832Z Running tests... 2023-01-11T21:44:22.7421092Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7421398Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7421661Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7421879Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20952 2023-01-11T21:44:22.7422093Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20953 2023-01-11T21:44:22.7422440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7422614Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7422990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7423177Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7423534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7423705Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7424073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7424260Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7424483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7424723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7425118Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7425571Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7425797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7426022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7426360Z STAGE:2023-01-11 21:36:29 20953:20953 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7426679Z STAGE:2023-01-11 21:36:29 20952:20952 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7427008Z STAGE:2023-01-11 21:36:29 20952:20952 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7427334Z STAGE:2023-01-11 21:36:29 20952:20952 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7427673Z STAGE:2023-01-11 21:36:29 20953:20953 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7428015Z STAGE:2023-01-11 21:36:29 20953:20953 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7428118Z ok (4.359s) 2023-01-11T21:44:22.7428138Z 2023-01-11T21:44:22.7428407Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7428519Z Ran 1 test in 4.359s 2023-01-11T21:44:22.7428538Z 2023-01-11T21:44:22.7428678Z OK 2023-01-11T21:44:22.7428697Z 2023-01-11T21:44:22.7428826Z Generating XML reports... 2023-01-11T21:44:22.7429274Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213625.xml 2023-01-11T21:44:22.7429619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7429793Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7430171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7430359Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7430378Z 2023-01-11T21:44:22.7430488Z Running tests... 2023-01-11T21:44:22.7430751Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7431066Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7431346Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7431545Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21065 2023-01-11T21:44:22.7431760Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21066 2023-01-11T21:44:22.7432126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7432302Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7432676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7432864Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7433227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7433401Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7433776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7433943Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7434183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7434635Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7434873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7435265Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7435490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7435718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7435960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T21:44:22.7436201Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T21:44:22.7436575Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.7436968Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.7437205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2023-01-11T21:44:22.7437441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2023-01-11T21:44:22.7437881Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2023-01-11T21:44:22.7438276Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2023-01-11T21:44:22.7438506Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms 2023-01-11T21:44:22.7438609Z ok (22.865s) 2023-01-11T21:44:22.7438628Z 2023-01-11T21:44:22.7438897Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7438996Z Ran 1 test in 22.865s 2023-01-11T21:44:22.7439016Z 2023-01-11T21:44:22.7439105Z OK 2023-01-11T21:44:22.7439124Z 2023-01-11T21:44:22.7439246Z Generating XML reports... 2023-01-11T21:44:22.7439691Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213632.xml 2023-01-11T21:44:22.7440057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7440233Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7440609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7440797Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7440817Z 2023-01-11T21:44:22.7440926Z Running tests... 2023-01-11T21:44:22.7441173Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7441487Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7441781Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7441999Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21192 2023-01-11T21:44:22.7442216Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21193 2023-01-11T21:44:22.7442584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7442756Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7443131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7443300Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7443723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7443893Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7444268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7444454Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7444697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7444938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7445331Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7445722Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7445938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7446161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7446395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T21:44:22.7446632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T21:44:22.7447069Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.7447461Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.7447695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2023-01-11T21:44:22.7447926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2023-01-11T21:44:22.7448309Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2023-01-11T21:44:22.7448675Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2023-01-11T21:44:22.7448902Z [E ProcessGroupGloo.cpp:2803] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms 2023-01-11T21:44:22.7449127Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 100 ms 2023-01-11T21:44:22.7449231Z ok (22.881s) 2023-01-11T21:44:22.7449251Z 2023-01-11T21:44:22.7449521Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7449635Z Ran 1 test in 22.881s 2023-01-11T21:44:22.7449655Z 2023-01-11T21:44:22.7449747Z OK 2023-01-11T21:44:22.7449766Z 2023-01-11T21:44:22.7449888Z Generating XML reports... 2023-01-11T21:44:22.7450486Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213657.xml 2023-01-11T21:44:22.7450878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7451051Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7451430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7451621Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7451640Z 2023-01-11T21:44:22.7451749Z Running tests... 2023-01-11T21:44:22.7452020Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7452328Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7452603Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7452898Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21319 2023-01-11T21:44:22.7453114Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21320 2023-01-11T21:44:22.7453486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7453657Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7454034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7454224Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7454583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7454755Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7455103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7455295Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7455535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7455772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7456227Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7456631Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7456855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7457079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7457235Z skip: Skipped due to small world size. (4.218s) 2023-01-11T21:44:22.7457258Z 2023-01-11T21:44:22.7457506Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7457620Z Ran 1 test in 4.218s 2023-01-11T21:44:22.7457639Z 2023-01-11T21:44:22.7457745Z OK (skipped=1) 2023-01-11T21:44:22.7457763Z 2023-01-11T21:44:22.7457883Z Generating XML reports... 2023-01-11T21:44:22.7458329Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213723.xml 2023-01-11T21:44:22.7458699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7458872Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7459246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7459434Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7459456Z 2023-01-11T21:44:22.7459546Z Running tests... 2023-01-11T21:44:22.7459810Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7460119Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7460380Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7460595Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21428 2023-01-11T21:44:22.7460812Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21429 2023-01-11T21:44:22.7461181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7461351Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7461707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7461950Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7462310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7462479Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7462845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7463033Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7463277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7463514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7463907Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7464282Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7464509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7464733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7464958Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms 2023-01-11T21:44:22.7465061Z ok (6.371s) 2023-01-11T21:44:22.7465127Z 2023-01-11T21:44:22.7465407Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7465520Z Ran 1 test in 6.371s 2023-01-11T21:44:22.7465539Z 2023-01-11T21:44:22.7465636Z OK 2023-01-11T21:44:22.7465655Z 2023-01-11T21:44:22.7465778Z Generating XML reports... 2023-01-11T21:44:22.7466205Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213729.xml 2023-01-11T21:44:22.7466576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7466749Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7467124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7467310Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7467330Z 2023-01-11T21:44:22.7467441Z Running tests... 2023-01-11T21:44:22.7467704Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7468009Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7468268Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7468482Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21537 2023-01-11T21:44:22.7468698Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21538 2023-01-11T21:44:22.7469063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7469232Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7469609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7469799Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7470164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7470335Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7470686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7470872Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7471172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7471414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7471809Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7472203Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7472428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7472653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7472891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T21:44:22.7473114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T21:44:22.7473513Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.7473948Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.7474163Z [E ProcessGroupGloo.cpp:138] Rank 0 timed out in monitoredBarrier after 0 ms. 2023-01-11T21:44:22.7474385Z No ranks successfully processed in monitoredBarrier. 2023-01-11T21:44:22.7474613Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 0 ms 2023-01-11T21:44:22.7474717Z ok (4.219s) 2023-01-11T21:44:22.7474737Z 2023-01-11T21:44:22.7475004Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7475098Z Ran 1 test in 4.220s 2023-01-11T21:44:22.7475117Z 2023-01-11T21:44:22.7475210Z OK 2023-01-11T21:44:22.7475233Z 2023-01-11T21:44:22.7475356Z Generating XML reports... 2023-01-11T21:44:22.7475803Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213738.xml 2023-01-11T21:44:22.7476168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7476343Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7476721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7476908Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7476927Z 2023-01-11T21:44:22.7477035Z Running tests... 2023-01-11T21:44:22.7477282Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7477590Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7477868Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7478084Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21652 2023-01-11T21:44:22.7478296Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21653 2023-01-11T21:44:22.7478663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7478838Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7479213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7479382Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7479743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7479909Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7480340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7480527Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7480769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7481013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7481407Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7481796Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7482004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7482227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7482468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T21:44:22.7482704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T21:44:22.7483098Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.7483530Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.7483767Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms 2023-01-11T21:44:22.7483870Z ok (4.468s) 2023-01-11T21:44:22.7483889Z 2023-01-11T21:44:22.7484156Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7484250Z Ran 1 test in 4.468s 2023-01-11T21:44:22.7484269Z 2023-01-11T21:44:22.7484366Z OK 2023-01-11T21:44:22.7484386Z 2023-01-11T21:44:22.7484510Z Generating XML reports... 2023-01-11T21:44:22.7484960Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213745.xml 2023-01-11T21:44:22.7485325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7485500Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7485878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7486069Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7486088Z 2023-01-11T21:44:22.7486178Z Running tests... 2023-01-11T21:44:22.7486443Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7486748Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7487024Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7487239Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21767 2023-01-11T21:44:22.7487450Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21768 2023-01-11T21:44:22.7487818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7487993Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7488368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7488538Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7488898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7489123Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7489494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7489680Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7489924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7490168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7490743Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7491120Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7491346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7491577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7491734Z skip: Skipped due to small world size. (4.156s) 2023-01-11T21:44:22.7491754Z 2023-01-11T21:44:22.7492022Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7492135Z Ran 1 test in 4.156s 2023-01-11T21:44:22.7492154Z 2023-01-11T21:44:22.7492262Z OK (skipped=1) 2023-01-11T21:44:22.7492280Z 2023-01-11T21:44:22.7492404Z Generating XML reports... 2023-01-11T21:44:22.7492927Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213752.xml 2023-01-11T21:44:22.7493285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7493460Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7493832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7494024Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7494044Z 2023-01-11T21:44:22.7494152Z Running tests... 2023-01-11T21:44:22.7494419Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7494729Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7495127Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s) 2023-01-11T21:44:22.7495147Z 2023-01-11T21:44:22.7495409Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7495503Z Ran 1 test in 0.002s 2023-01-11T21:44:22.7495522Z 2023-01-11T21:44:22.7495630Z OK (skipped=1) 2023-01-11T21:44:22.7495649Z 2023-01-11T21:44:22.7495772Z Generating XML reports... 2023-01-11T21:44:22.7496214Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213759.xml 2023-01-11T21:44:22.7496581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7496753Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7497125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7497316Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7497335Z 2023-01-11T21:44:22.7497443Z Running tests... 2023-01-11T21:44:22.7497688Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7497998Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7498390Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s) 2023-01-11T21:44:22.7498474Z 2023-01-11T21:44:22.7498741Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7498852Z Ran 1 test in 0.003s 2023-01-11T21:44:22.7498871Z 2023-01-11T21:44:22.7498980Z OK (skipped=1) 2023-01-11T21:44:22.7498998Z 2023-01-11T21:44:22.7499122Z Generating XML reports... 2023-01-11T21:44:22.7499558Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213801.xml 2023-01-11T21:44:22.7499905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7500078Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7500445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7500629Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7500648Z 2023-01-11T21:44:22.7500756Z Running tests... 2023-01-11T21:44:22.7501016Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7501320Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7501714Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s) 2023-01-11T21:44:22.7501734Z 2023-01-11T21:44:22.7502036Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7502133Z Ran 1 test in 0.002s 2023-01-11T21:44:22.7502170Z 2023-01-11T21:44:22.7502259Z OK (skipped=1) 2023-01-11T21:44:22.7502278Z 2023-01-11T21:44:22.7502401Z Generating XML reports... 2023-01-11T21:44:22.7502843Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213804.xml 2023-01-11T21:44:22.7503206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7503379Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7503750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7503937Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7503956Z 2023-01-11T21:44:22.7504064Z Running tests... 2023-01-11T21:44:22.7504312Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7504619Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7505010Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.003s) 2023-01-11T21:44:22.7505030Z 2023-01-11T21:44:22.7505289Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7505400Z Ran 1 test in 0.003s 2023-01-11T21:44:22.7505423Z 2023-01-11T21:44:22.7505532Z OK (skipped=1) 2023-01-11T21:44:22.7505550Z 2023-01-11T21:44:22.7505671Z Generating XML reports... 2023-01-11T21:44:22.7506109Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213806.xml 2023-01-11T21:44:22.7506472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7506628Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7506996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7507181Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7507200Z 2023-01-11T21:44:22.7507307Z Running tests... 2023-01-11T21:44:22.7507616Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7507926Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7508283Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL backend supports high priority stream (0.002s) 2023-01-11T21:44:22.7508302Z 2023-01-11T21:44:22.7508565Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7508677Z Ran 1 test in 0.002s 2023-01-11T21:44:22.7508696Z 2023-01-11T21:44:22.7508786Z OK (skipped=1) 2023-01-11T21:44:22.7508804Z 2023-01-11T21:44:22.7508930Z Generating XML reports... 2023-01-11T21:44:22.7509366Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213809.xml 2023-01-11T21:44:22.7509728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7509898Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7510266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7510457Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7510476Z 2023-01-11T21:44:22.7510586Z Running tests... 2023-01-11T21:44:22.7510831Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7511137Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7511429Z test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2023-01-11T21:44:22.7511450Z 2023-01-11T21:44:22.7511715Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7511827Z Ran 1 test in 0.002s 2023-01-11T21:44:22.7511846Z 2023-01-11T21:44:22.7511952Z OK (skipped=1) 2023-01-11T21:44:22.7511971Z 2023-01-11T21:44:22.7512097Z Generating XML reports... 2023-01-11T21:44:22.7512543Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213811.xml 2023-01-11T21:44:22.7512915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7513069Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7513443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7513634Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7513653Z 2023-01-11T21:44:22.7513761Z Running tests... 2023-01-11T21:44:22.7514026Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7514332Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7514598Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2023-01-11T21:44:22.7514621Z 2023-01-11T21:44:22.7514880Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7514991Z Ran 1 test in 0.002s 2023-01-11T21:44:22.7515010Z 2023-01-11T21:44:22.7515099Z OK (skipped=1) 2023-01-11T21:44:22.7515118Z 2023-01-11T21:44:22.7515241Z Generating XML reports... 2023-01-11T21:44:22.7515682Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213813.xml 2023-01-11T21:44:22.7516047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7516222Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7516596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7516783Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7516802Z 2023-01-11T21:44:22.7516982Z Running tests... 2023-01-11T21:44:22.7517227Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7517538Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7517842Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2023-01-11T21:44:22.7517862Z 2023-01-11T21:44:22.7518123Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7518236Z Ran 1 test in 0.002s 2023-01-11T21:44:22.7518255Z 2023-01-11T21:44:22.7518364Z OK (skipped=1) 2023-01-11T21:44:22.7518382Z 2023-01-11T21:44:22.7518504Z Generating XML reports... 2023-01-11T21:44:22.7518945Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213816.xml 2023-01-11T21:44:22.7519312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7519470Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7519848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7520036Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7520055Z 2023-01-11T21:44:22.7520161Z Running tests... 2023-01-11T21:44:22.7520524Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7520838Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7521137Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7521352Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22140 2023-01-11T21:44:22.7521566Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22141 2023-01-11T21:44:22.7521919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7522092Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7522463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7522652Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7523008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7523179Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7523552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7523740Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7523965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7524205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7524601Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7524990Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7525215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7525439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7525540Z ok (4.121s) 2023-01-11T21:44:22.7525559Z 2023-01-11T21:44:22.7525825Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7525939Z Ran 1 test in 4.121s 2023-01-11T21:44:22.7526006Z 2023-01-11T21:44:22.7526082Z OK 2023-01-11T21:44:22.7526100Z 2023-01-11T21:44:22.7526228Z Generating XML reports... 2023-01-11T21:44:22.7526674Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213818.xml 2023-01-11T21:44:22.7527040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7527217Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7527590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7527778Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7527797Z 2023-01-11T21:44:22.7527904Z Running tests... 2023-01-11T21:44:22.7528166Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7528458Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7528748Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7528961Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22249 2023-01-11T21:44:22.7529172Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22250 2023-01-11T21:44:22.7529582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7529761Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7530136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7530488Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7530844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7531022Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7531392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7531574Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7531811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7532053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7532447Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7532836Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7533064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7533272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7533370Z ok (4.162s) 2023-01-11T21:44:22.7533390Z 2023-01-11T21:44:22.7533657Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7533767Z Ran 1 test in 4.162s 2023-01-11T21:44:22.7533787Z 2023-01-11T21:44:22.7533879Z OK 2023-01-11T21:44:22.7533898Z 2023-01-11T21:44:22.7534035Z Generating XML reports... 2023-01-11T21:44:22.7534481Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213825.xml 2023-01-11T21:44:22.7534846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7535000Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7535372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7535643Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7535662Z 2023-01-11T21:44:22.7535774Z Running tests... 2023-01-11T21:44:22.7536038Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7536345Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7536615Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2023-01-11T21:44:22.7536635Z 2023-01-11T21:44:22.7536897Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7537004Z Ran 1 test in 0.002s 2023-01-11T21:44:22.7537023Z 2023-01-11T21:44:22.7537113Z OK (skipped=1) 2023-01-11T21:44:22.7537148Z 2023-01-11T21:44:22.7537252Z Generating XML reports... 2023-01-11T21:44:22.7537695Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213832.xml 2023-01-11T21:44:22.7538061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7538232Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7538603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7538849Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7538870Z 2023-01-11T21:44:22.7538985Z Running tests... 2023-01-11T21:44:22.7539251Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7539542Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7539834Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2023-01-11T21:44:22.7539858Z 2023-01-11T21:44:22.7540113Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7540223Z Ran 1 test in 0.002s 2023-01-11T21:44:22.7540243Z 2023-01-11T21:44:22.7540347Z OK (skipped=1) 2023-01-11T21:44:22.7540366Z 2023-01-11T21:44:22.7540487Z Generating XML reports... 2023-01-11T21:44:22.7540929Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213834.xml 2023-01-11T21:44:22.7541297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7541471Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7541826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7542014Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7542037Z 2023-01-11T21:44:22.7542143Z Running tests... 2023-01-11T21:44:22.7542406Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7542714Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7542986Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7543731Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.672s) 2023-01-11T21:44:22.7543752Z 2023-01-11T21:44:22.7544011Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7544122Z Ran 1 test in 1.672s 2023-01-11T21:44:22.7544141Z 2023-01-11T21:44:22.7544229Z OK (skipped=1) 2023-01-11T21:44:22.7544319Z 2023-01-11T21:44:22.7544428Z Generating XML reports... 2023-01-11T21:44:22.7544875Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213837.xml 2023-01-11T21:44:22.7545239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7545411Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7545789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7545975Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7545994Z 2023-01-11T21:44:22.7546101Z Running tests... 2023-01-11T21:44:22.7546363Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7546651Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7546929Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7547142Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22458 2023-01-11T21:44:22.7547356Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22459 2023-01-11T21:44:22.7547722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7547941Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7548319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7548506Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7548860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7549015Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7549380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7549563Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7549801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7550040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7550433Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7550822Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7551045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7551255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7551354Z ok (6.249s) 2023-01-11T21:44:22.7551374Z 2023-01-11T21:44:22.7551635Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7551747Z Ran 1 test in 6.249s 2023-01-11T21:44:22.7551766Z 2023-01-11T21:44:22.7551857Z OK 2023-01-11T21:44:22.7551876Z 2023-01-11T21:44:22.7551998Z Generating XML reports... 2023-01-11T21:44:22.7552447Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213841.xml 2023-01-11T21:44:22.7552809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7552981Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7553333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7553567Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7553586Z 2023-01-11T21:44:22.7553690Z Running tests... 2023-01-11T21:44:22.7553955Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7554261Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7554523Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7554741Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22573 2023-01-11T21:44:22.7554955Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22574 2023-01-11T21:44:22.7555303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7555473Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7555839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7556027Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7556385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7556554Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7556962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7557148Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7557390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7557770Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7558009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7558397Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7558622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7558845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7558947Z ok (6.968s) 2023-01-11T21:44:22.7558970Z 2023-01-11T21:44:22.7559236Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7559347Z Ran 1 test in 6.968s 2023-01-11T21:44:22.7559366Z 2023-01-11T21:44:22.7559441Z OK 2023-01-11T21:44:22.7559477Z 2023-01-11T21:44:22.7559581Z Generating XML reports... 2023-01-11T21:44:22.7560026Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213849.xml 2023-01-11T21:44:22.7560396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7560568Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7560940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7561124Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7561143Z 2023-01-11T21:44:22.7561254Z Running tests... 2023-01-11T21:44:22.7561516Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7561808Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7562089Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7562303Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22685 2023-01-11T21:44:22.7562569Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22686 2023-01-11T21:44:22.7562936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7563108Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7563476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7563664Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7564019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7564171Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7564530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7564708Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7564950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7565343Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7565572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7566005Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7566236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7566440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7566543Z ok (6.926s) 2023-01-11T21:44:22.7566562Z 2023-01-11T21:44:22.7566828Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7566942Z Ran 1 test in 6.926s 2023-01-11T21:44:22.7566961Z 2023-01-11T21:44:22.7567050Z OK 2023-01-11T21:44:22.7567069Z 2023-01-11T21:44:22.7567185Z Generating XML reports... 2023-01-11T21:44:22.7567629Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213859.xml 2023-01-11T21:44:22.7567996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7568173Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7568529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7568716Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7568735Z 2023-01-11T21:44:22.7568842Z Running tests... 2023-01-11T21:44:22.7569101Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7569411Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7569684Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7570592Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.635s) 2023-01-11T21:44:22.7570614Z 2023-01-11T21:44:22.7570893Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7570996Z Ran 1 test in 1.636s 2023-01-11T21:44:22.7571017Z 2023-01-11T21:44:22.7571106Z OK (skipped=1) 2023-01-11T21:44:22.7571141Z 2023-01-11T21:44:22.7571248Z Generating XML reports... 2023-01-11T21:44:22.7571689Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213908.xml 2023-01-11T21:44:22.7572142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7572320Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7572683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7572871Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7572891Z 2023-01-11T21:44:22.7572995Z Running tests... 2023-01-11T21:44:22.7573253Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7573712Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7574039Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7574804Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.657s) 2023-01-11T21:44:22.7574812Z 2023-01-11T21:44:22.7575074Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7575180Z Ran 1 test in 1.657s 2023-01-11T21:44:22.7575199Z 2023-01-11T21:44:22.7575381Z OK (skipped=1) 2023-01-11T21:44:22.7575402Z 2023-01-11T21:44:22.7575531Z Generating XML reports... 2023-01-11T21:44:22.7575988Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213913.xml 2023-01-11T21:44:22.7576343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7576512Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7576872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7577059Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7577079Z 2023-01-11T21:44:22.7577184Z Running tests... 2023-01-11T21:44:22.7577465Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7578009Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7578294Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7578509Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22865 2023-01-11T21:44:22.7578724Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22866 2023-01-11T21:44:22.7579075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7579249Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7579623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7579809Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7580170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7580336Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7580700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7580885Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7581125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7581431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7581830Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7582220Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7582448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7582671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7582819Z skip: Need at least 4 CUDA devices (4.148s) 2023-01-11T21:44:22.7582839Z 2023-01-11T21:44:22.7583099Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7583204Z Ran 1 test in 4.148s 2023-01-11T21:44:22.7583224Z 2023-01-11T21:44:22.7583329Z OK (skipped=1) 2023-01-11T21:44:22.7583352Z 2023-01-11T21:44:22.7583457Z Generating XML reports... 2023-01-11T21:44:22.7583906Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213917.xml 2023-01-11T21:44:22.7584267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7584440Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7584858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7585050Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7585069Z 2023-01-11T21:44:22.7585178Z Running tests... 2023-01-11T21:44:22.7585439Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7585731Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7586052Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7586262Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22974 2023-01-11T21:44:22.7586470Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22975 2023-01-11T21:44:22.7586838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7587012Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7587373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7587537Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7587911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7588083Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7588456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7588638Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7588877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7589120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7589512Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7589901Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7590127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7590410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7590542Z skip: Need at least 4 CUDA devices (4.264s) 2023-01-11T21:44:22.7590562Z 2023-01-11T21:44:22.7590831Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7590942Z Ran 1 test in 4.264s 2023-01-11T21:44:22.7590962Z 2023-01-11T21:44:22.7591068Z OK (skipped=1) 2023-01-11T21:44:22.7591087Z 2023-01-11T21:44:22.7591209Z Generating XML reports... 2023-01-11T21:44:22.7591660Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213924.xml 2023-01-11T21:44:22.7592020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7592196Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7592552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7592741Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7592760Z 2023-01-11T21:44:22.7592864Z Running tests... 2023-01-11T21:44:22.7593128Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7593439Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7593781Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7594534Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84886 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.646s) 2023-01-11T21:44:22.7594555Z 2023-01-11T21:44:22.7594819Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7594934Z Ran 1 test in 1.646s 2023-01-11T21:44:22.7594954Z 2023-01-11T21:44:22.7595057Z OK (skipped=1) 2023-01-11T21:44:22.7595076Z 2023-01-11T21:44:22.7595181Z Generating XML reports... 2023-01-11T21:44:22.7595626Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213930.xml 2023-01-11T21:44:22.7595991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7596170Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7596546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7596734Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7596753Z 2023-01-11T21:44:22.7596860Z Running tests... 2023-01-11T21:44:22.7597123Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7597437Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7597678Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7597894Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23117 2023-01-11T21:44:22.7598105Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23118 2023-01-11T21:44:22.7598471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7598644Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7599015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7599198Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7599621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7599771Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7600134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7600315Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7600557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7600796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7601190Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7601579Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7601807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7602032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7602249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T21:44:22.7602484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T21:44:22.7602925Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.7603323Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.7603655Z STAGE:2023-01-11 21:39:38 23117:23117 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7603976Z STAGE:2023-01-11 21:39:38 23118:23118 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7604518Z STAGE:2023-01-11 21:39:38 23117:23117 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:39:38 23118:23118 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7604539Z 2023-01-11T21:44:22.7605102Z STAGE:2023-01-11 21:39:38 23117:23117 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:39:38 23118:23118 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7605122Z 2023-01-11T21:44:22.7605446Z STAGE:2023-01-11 21:39:38 23118:23118 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7605760Z STAGE:2023-01-11 21:39:38 23117:23117 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7606087Z STAGE:2023-01-11 21:39:38 23117:23117 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7606400Z STAGE:2023-01-11 21:39:38 23118:23118 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7606744Z STAGE:2023-01-11 21:39:38 23117:23117 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7607080Z STAGE:2023-01-11 21:39:38 23118:23118 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7607179Z ok (4.251s) 2023-01-11T21:44:22.7607198Z 2023-01-11T21:44:22.7607461Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7607572Z Ran 1 test in 4.251s 2023-01-11T21:44:22.7607591Z 2023-01-11T21:44:22.7607682Z OK 2023-01-11T21:44:22.7607701Z 2023-01-11T21:44:22.7607820Z Generating XML reports... 2023-01-11T21:44:22.7608252Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213935.xml 2023-01-11T21:44:22.7608621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7608849Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7609230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7609416Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7609436Z 2023-01-11T21:44:22.7609543Z Running tests... 2023-01-11T21:44:22.7609804Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7610118Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7610553Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7610762Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23236 2023-01-11T21:44:22.7610973Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23237 2023-01-11T21:44:22.7611352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7611523Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7611893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7612076Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7612509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7612687Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7613041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7613221Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7613462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7613704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7614093Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7614480Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7614705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7614918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7615153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T21:44:22.7615370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T21:44:22.7615762Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.7616155Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.7616684Z STAGE:2023-01-11 21:39:45 23236:23236 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:39:45 23237:23237 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7616705Z 2023-01-11T21:44:22.7617036Z STAGE:2023-01-11 21:39:45 23237:23237 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7617360Z STAGE:2023-01-11 21:39:45 23236:23236 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7617916Z STAGE:2023-01-11 21:39:45 23236:23236 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:39:45 23237:23237 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7618000Z 2023-01-11T21:44:22.7618331Z STAGE:2023-01-11 21:39:45 23237:23237 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7618645Z STAGE:2023-01-11 21:39:45 23236:23236 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7618968Z STAGE:2023-01-11 21:39:45 23236:23236 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7619517Z STAGE:2023-01-11 21:39:45 23236:23236 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:39:45 23237:23237 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7619538Z 2023-01-11T21:44:22.7619879Z STAGE:2023-01-11 21:39:45 23237:23237 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7619964Z ok (4.368s) 2023-01-11T21:44:22.7619983Z 2023-01-11T21:44:22.7620250Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7620361Z Ran 1 test in 4.369s 2023-01-11T21:44:22.7620383Z 2023-01-11T21:44:22.7620471Z OK 2023-01-11T21:44:22.7620490Z 2023-01-11T21:44:22.7620611Z Generating XML reports... 2023-01-11T21:44:22.7621056Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213941.xml 2023-01-11T21:44:22.7621421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7621643Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7622007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7622197Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7622216Z 2023-01-11T21:44:22.7622322Z Running tests... 2023-01-11T21:44:22.7622583Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7622890Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7623152Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7623363Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23355 2023-01-11T21:44:22.7623574Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23356 2023-01-11T21:44:22.7623943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7624099Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7624472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7624659Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7625018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7625186Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7625551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7625730Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7625974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7626193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7626591Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7626979Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7627202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7627479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7627710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T21:44:22.7627943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T21:44:22.7628337Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.7628669Z STAGE:2023-01-11 21:39:52 23356:23356 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7629040Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.7629367Z STAGE:2023-01-11 21:39:52 23355:23355 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7629697Z STAGE:2023-01-11 21:39:52 23356:23356 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7630024Z STAGE:2023-01-11 21:39:52 23355:23355 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7630369Z STAGE:2023-01-11 21:39:52 23356:23356 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7630706Z STAGE:2023-01-11 21:39:52 23355:23355 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7631070Z STAGE:2023-01-11 21:39:52 23356:23356 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7631389Z STAGE:2023-01-11 21:39:52 23355:23355 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7631712Z STAGE:2023-01-11 21:39:52 23355:23355 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7632259Z STAGE:2023-01-11 21:39:52 23356:23356 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:39:52 23355:23355 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7632283Z 2023-01-11T21:44:22.7632603Z STAGE:2023-01-11 21:39:52 23356:23356 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7632705Z ok (4.321s) 2023-01-11T21:44:22.7632724Z 2023-01-11T21:44:22.7632989Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7633100Z Ran 1 test in 4.322s 2023-01-11T21:44:22.7633120Z 2023-01-11T21:44:22.7633215Z OK 2023-01-11T21:44:22.7633234Z 2023-01-11T21:44:22.7633357Z Generating XML reports... 2023-01-11T21:44:22.7633808Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213948.xml 2023-01-11T21:44:22.7634175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7634349Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7634713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7634899Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7634918Z 2023-01-11T21:44:22.7635026Z Running tests... 2023-01-11T21:44:22.7635290Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7635599Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7635857Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7636071Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23474 2023-01-11T21:44:22.7636278Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23475 2023-01-11T21:44:22.7636626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7636853Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7637225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7637408Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7637771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7637943Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7638318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7638501Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7638740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7638961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7639362Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7639756Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7639973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7640244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7640486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T21:44:22.7640722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T21:44:22.7641118Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.7641514Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.7641828Z STAGE:2023-01-11 21:39:59 23474:23474 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7642142Z STAGE:2023-01-11 21:39:59 23475:23475 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7642681Z STAGE:2023-01-11 21:39:59 23474:23474 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:39:59 23475:23475 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7642702Z 2023-01-11T21:44:22.7643259Z STAGE:2023-01-11 21:39:59 23474:23474 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:39:59 23475:23475 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7643279Z 2023-01-11T21:44:22.7643603Z STAGE:2023-01-11 21:39:59 23475:23475 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7643922Z STAGE:2023-01-11 21:39:59 23474:23474 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7644248Z STAGE:2023-01-11 21:39:59 23474:23474 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7644570Z STAGE:2023-01-11 21:39:59 23475:23475 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7644912Z STAGE:2023-01-11 21:39:59 23474:23474 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7645251Z STAGE:2023-01-11 21:39:59 23475:23475 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7645336Z ok (4.248s) 2023-01-11T21:44:22.7645368Z 2023-01-11T21:44:22.7645615Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7645723Z Ran 1 test in 4.249s 2023-01-11T21:44:22.7645742Z 2023-01-11T21:44:22.7645831Z OK 2023-01-11T21:44:22.7645913Z 2023-01-11T21:44:22.7646038Z Generating XML reports... 2023-01-11T21:44:22.7646486Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213955.xml 2023-01-11T21:44:22.7646852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7647025Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7647405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7647577Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7647597Z 2023-01-11T21:44:22.7647702Z Running tests... 2023-01-11T21:44:22.7647964Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7648275Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7648532Z test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7648748Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23593 2023-01-11T21:44:22.7648961Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23594 2023-01-11T21:44:22.7649327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7649527Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7649912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7650097Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7650674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7650852Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7667168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7667423Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7667681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7667939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7668382Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7668795Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7669034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7669269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7669436Z skip: Skipped due to small world size. (4.267s) 2023-01-11T21:44:22.7669458Z 2023-01-11T21:44:22.7669743Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7669857Z Ran 1 test in 4.267s 2023-01-11T21:44:22.7669877Z 2023-01-11T21:44:22.7669986Z OK (skipped=1) 2023-01-11T21:44:22.7670005Z 2023-01-11T21:44:22.7670129Z Generating XML reports... 2023-01-11T21:44:22.7670590Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214002.xml 2023-01-11T21:44:22.7670973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7671142Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7671535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7671879Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7671900Z 2023-01-11T21:44:22.7671999Z Running tests... 2023-01-11T21:44:22.7672276Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7672596Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7672846Z test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7673063Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23702 2023-01-11T21:44:22.7673276Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23703 2023-01-11T21:44:22.7673653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7673884Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7674284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7674469Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7674846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7675016Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7675475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7675671Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7675911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7676150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7676561Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7676964Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7677189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7677418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7677568Z skip: Skipped due to small world size. (4.255s) 2023-01-11T21:44:22.7677595Z 2023-01-11T21:44:22.7677875Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7677969Z Ran 1 test in 4.256s 2023-01-11T21:44:22.7677989Z 2023-01-11T21:44:22.7678093Z OK (skipped=1) 2023-01-11T21:44:22.7678116Z 2023-01-11T21:44:22.7678226Z Generating XML reports... 2023-01-11T21:44:22.7678688Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214009.xml 2023-01-11T21:44:22.7679068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7679237Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7679622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7679803Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7679829Z 2023-01-11T21:44:22.7679922Z Running tests... 2023-01-11T21:44:22.7680194Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7680511Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7680772Z test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7680988Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23811 2023-01-11T21:44:22.7681315Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23812 2023-01-11T21:44:22.7681696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7681867Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7682253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7682440Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7682815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7682984Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7683371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7683558Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7683800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7684038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7684435Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7684885Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7685121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7685346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7685499Z skip: Skipped due to small world size. (4.269s) 2023-01-11T21:44:22.7685520Z 2023-01-11T21:44:22.7685800Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7685905Z Ran 1 test in 4.269s 2023-01-11T21:44:22.7685925Z 2023-01-11T21:44:22.7686023Z OK (skipped=1) 2023-01-11T21:44:22.7686043Z 2023-01-11T21:44:22.7686158Z Generating XML reports... 2023-01-11T21:44:22.7686613Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214016.xml 2023-01-11T21:44:22.7686994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7687166Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7687555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7687741Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7687761Z 2023-01-11T21:44:22.7687861Z Running tests... 2023-01-11T21:44:22.7688135Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7688454Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7688707Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7688917Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23920 2023-01-11T21:44:22.7689135Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23921 2023-01-11T21:44:22.7689511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7689681Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7690070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7690255Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7690944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7691115Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7691498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7691685Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7691930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7692172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7692578Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7692982Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7693212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7693438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7693590Z skip: Skipped due to small world size. (4.228s) 2023-01-11T21:44:22.7693612Z 2023-01-11T21:44:22.7693877Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7694053Z Ran 1 test in 4.228s 2023-01-11T21:44:22.7694075Z 2023-01-11T21:44:22.7694179Z OK (skipped=1) 2023-01-11T21:44:22.7694199Z 2023-01-11T21:44:22.7694315Z Generating XML reports... 2023-01-11T21:44:22.7694780Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214022.xml 2023-01-11T21:44:22.7695158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7695332Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7695718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7695905Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7695925Z 2023-01-11T21:44:22.7696019Z Running tests... 2023-01-11T21:44:22.7696288Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7696609Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7696852Z test_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7697067Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24029 2023-01-11T21:44:22.7697284Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24030 2023-01-11T21:44:22.7697663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7697837Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7698220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7698404Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7698781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7698950Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7699338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7699522Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7699774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7700102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7700522Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7700916Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7701160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7701397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7701744Z STAGE:2023-01-11 21:40:33 24029:24029 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7702084Z STAGE:2023-01-11 21:40:33 24030:24030 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7702655Z STAGE:2023-01-11 21:40:33 24029:24029 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:40:33 24030:24030 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7702679Z 2023-01-11T21:44:22.7703279Z STAGE:2023-01-11 21:40:33 24029:24029 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:40:33 24030:24030 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7703299Z 2023-01-11T21:44:22.7703689Z STAGE:2023-01-11 21:40:33 24030:24030 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7704042Z STAGE:2023-01-11 21:40:33 24029:24029 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7704389Z STAGE:2023-01-11 21:40:33 24029:24029 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7704720Z STAGE:2023-01-11 21:40:33 24030:24030 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7705080Z STAGE:2023-01-11 21:40:33 24029:24029 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7705444Z STAGE:2023-01-11 21:40:33 24030:24030 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7705545Z ok (4.234s) 2023-01-11T21:44:22.7705564Z 2023-01-11T21:44:22.7705843Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7705960Z Ran 1 test in 4.234s 2023-01-11T21:44:22.7705979Z 2023-01-11T21:44:22.7706069Z OK 2023-01-11T21:44:22.7706092Z 2023-01-11T21:44:22.7706219Z Generating XML reports... 2023-01-11T21:44:22.7706693Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214029.xml 2023-01-11T21:44:22.7707064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7707244Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7707648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7707846Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7707866Z 2023-01-11T21:44:22.7707974Z Running tests... 2023-01-11T21:44:22.7708253Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7708582Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7708842Z test_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7709053Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24142 2023-01-11T21:44:22.7709282Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24143 2023-01-11T21:44:22.7709670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7709906Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7710307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7710505Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7710892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7711077Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7711478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7711655Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7711908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7712164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7712587Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7713003Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7713243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7713526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7713883Z STAGE:2023-01-11 21:40:40 24142:24142 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7714226Z STAGE:2023-01-11 21:40:40 24143:24143 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7714557Z STAGE:2023-01-11 21:40:40 24143:24143 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7714908Z STAGE:2023-01-11 21:40:40 24142:24142 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7715280Z STAGE:2023-01-11 21:40:40 24143:24143 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7715645Z STAGE:2023-01-11 21:40:40 24142:24142 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7715984Z STAGE:2023-01-11 21:40:40 24143:24143 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7716324Z STAGE:2023-01-11 21:40:40 24142:24142 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7716894Z STAGE:2023-01-11 21:40:40 24143:24143 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:40:40 24142:24142 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7716915Z 2023-01-11T21:44:22.7717280Z STAGE:2023-01-11 21:40:40 24143:24143 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7717649Z STAGE:2023-01-11 21:40:40 24142:24142 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7717735Z ok (4.222s) 2023-01-11T21:44:22.7717773Z 2023-01-11T21:44:22.7718037Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7718152Z Ran 1 test in 4.223s 2023-01-11T21:44:22.7718171Z 2023-01-11T21:44:22.7718267Z OK 2023-01-11T21:44:22.7718286Z 2023-01-11T21:44:22.7718414Z Generating XML reports... 2023-01-11T21:44:22.7718887Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214036.xml 2023-01-11T21:44:22.7719277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7719460Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7719859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7720090Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7720110Z 2023-01-11T21:44:22.7720213Z Running tests... 2023-01-11T21:44:22.7720495Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7720820Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7721105Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports reduce multigpu (0.002s) 2023-01-11T21:44:22.7721125Z 2023-01-11T21:44:22.7721401Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7721513Z Ran 1 test in 0.002s 2023-01-11T21:44:22.7721532Z 2023-01-11T21:44:22.7721638Z OK (skipped=1) 2023-01-11T21:44:22.7721657Z 2023-01-11T21:44:22.7721781Z Generating XML reports... 2023-01-11T21:44:22.7722234Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214043.xml 2023-01-11T21:44:22.7722619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7722801Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7723197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7723391Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7723461Z 2023-01-11T21:44:22.7723575Z Running tests... 2023-01-11T21:44:22.7723857Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7724188Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7724431Z test_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7724658Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24288 2023-01-11T21:44:22.7724887Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24289 2023-01-11T21:44:22.7725272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7725450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7725845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7726039Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7726423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7726600Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7726981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7727176Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7727424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7727671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7728083Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7728494Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7728728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7728965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7729516Z STAGE:2023-01-11 21:40:49 24288:24288 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:40:49 24289:24289 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7729603Z 2023-01-11T21:44:22.7729948Z STAGE:2023-01-11 21:40:49 24289:24289 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7730480Z STAGE:2023-01-11 21:40:49 24288:24288 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7730872Z STAGE:2023-01-11 21:40:49 24289:24289 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7731236Z STAGE:2023-01-11 21:40:49 24288:24288 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7731577Z STAGE:2023-01-11 21:40:49 24289:24289 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7731919Z STAGE:2023-01-11 21:40:49 24288:24288 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7732483Z STAGE:2023-01-11 21:40:49 24288:24288 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:40:49 24289:24289 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7732509Z 2023-01-11T21:44:22.7732872Z STAGE:2023-01-11 21:40:49 24289:24289 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7733232Z STAGE:2023-01-11 21:40:49 24288:24288 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7733336Z ok (4.255s) 2023-01-11T21:44:22.7733356Z 2023-01-11T21:44:22.7733698Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7733824Z Ran 1 test in 4.255s 2023-01-11T21:44:22.7733844Z 2023-01-11T21:44:22.7733936Z OK 2023-01-11T21:44:22.7733955Z 2023-01-11T21:44:22.7734081Z Generating XML reports... 2023-01-11T21:44:22.7734556Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214045.xml 2023-01-11T21:44:22.7734947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7735133Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7735532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7735711Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7735750Z 2023-01-11T21:44:22.7735844Z Running tests... 2023-01-11T21:44:22.7736127Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7736462Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7736769Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce_scatter_tensor (0.002s) 2023-01-11T21:44:22.7736788Z 2023-01-11T21:44:22.7737066Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7737182Z Ran 1 test in 0.002s 2023-01-11T21:44:22.7737201Z 2023-01-11T21:44:22.7737304Z OK (skipped=1) 2023-01-11T21:44:22.7737323Z 2023-01-11T21:44:22.7737447Z Generating XML reports... 2023-01-11T21:44:22.7737899Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214052.xml 2023-01-11T21:44:22.7738288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7738472Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7738874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7739072Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7739091Z 2023-01-11T21:44:22.7739205Z Running tests... 2023-01-11T21:44:22.7739487Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7739889Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7740173Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports reduce_scatter_v (0.003s) 2023-01-11T21:44:22.7740193Z 2023-01-11T21:44:22.7740454Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7740569Z Ran 1 test in 0.003s 2023-01-11T21:44:22.7740589Z 2023-01-11T21:44:22.7740695Z OK (skipped=1) 2023-01-11T21:44:22.7740718Z 2023-01-11T21:44:22.7740846Z Generating XML reports... 2023-01-11T21:44:22.7741320Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214054.xml 2023-01-11T21:44:22.7741708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7741890Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7742294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7742490Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7742510Z 2023-01-11T21:44:22.7742604Z Running tests... 2023-01-11T21:44:22.7742886Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7743213Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7743512Z test_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7743749Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24467 2023-01-11T21:44:22.7743980Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24468 2023-01-11T21:44:22.7744370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7744555Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7744937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7745134Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7745519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7745701Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7746100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7746294Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7746544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7746799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7747219Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7747615Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7747852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7748093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7748440Z STAGE:2023-01-11 21:41:01 24467:24467 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7748784Z STAGE:2023-01-11 21:41:01 24468:24468 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7749133Z STAGE:2023-01-11 21:41:01 24467:24467 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7749495Z STAGE:2023-01-11 21:41:01 24468:24468 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7750137Z STAGE:2023-01-11 21:41:01 24467:24467 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:41:01 24468:24468 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7750171Z 2023-01-11T21:44:22.7750498Z STAGE:2023-01-11 21:41:01 24468:24468 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7750826Z STAGE:2023-01-11 21:41:01 24467:24467 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7751168Z STAGE:2023-01-11 21:41:01 24467:24467 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7751513Z STAGE:2023-01-11 21:41:01 24468:24468 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7751869Z STAGE:2023-01-11 21:41:01 24467:24467 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7752237Z STAGE:2023-01-11 21:41:01 24468:24468 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7752338Z ok (4.272s) 2023-01-11T21:44:22.7752357Z 2023-01-11T21:44:22.7752636Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7752751Z Ran 1 test in 4.272s 2023-01-11T21:44:22.7752770Z 2023-01-11T21:44:22.7752862Z OK 2023-01-11T21:44:22.7752881Z 2023-01-11T21:44:22.7752991Z Generating XML reports... 2023-01-11T21:44:22.7753509Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214057.xml 2023-01-11T21:44:22.7753908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7754090Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7754486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7754687Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7754707Z 2023-01-11T21:44:22.7754816Z Running tests... 2023-01-11T21:44:22.7755100Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7755426Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7755678Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s) 2023-01-11T21:44:22.7755698Z 2023-01-11T21:44:22.7755978Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7756094Z Ran 1 test in 0.002s 2023-01-11T21:44:22.7756113Z 2023-01-11T21:44:22.7756223Z OK (skipped=1) 2023-01-11T21:44:22.7756242Z 2023-01-11T21:44:22.7756367Z Generating XML reports... 2023-01-11T21:44:22.7756837Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214104.xml 2023-01-11T21:44:22.7757232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7757414Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7757798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7757997Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7758019Z 2023-01-11T21:44:22.7758127Z Running tests... 2023-01-11T21:44:22.7758411Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7758742Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7759017Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s) 2023-01-11T21:44:22.7759037Z 2023-01-11T21:44:22.7759317Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7759492Z Ran 1 test in 0.002s 2023-01-11T21:44:22.7759511Z 2023-01-11T21:44:22.7759622Z OK (skipped=1) 2023-01-11T21:44:22.7759641Z 2023-01-11T21:44:22.7759749Z Generating XML reports... 2023-01-11T21:44:22.7760222Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214106.xml 2023-01-11T21:44:22.7760613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7760793Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7761192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7761390Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7761409Z 2023-01-11T21:44:22.7761519Z Running tests... 2023-01-11T21:44:22.7761799Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7762131Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7762375Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7762603Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24646 2023-01-11T21:44:22.7762875Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24647 2023-01-11T21:44:22.7763272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7763456Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7763856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7764054Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7764448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7764613Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7765012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7765208Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7765462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7765880Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7766131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7766546Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7766786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7767022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7767352Z STAGE:2023-01-11 21:41:12 24646:24646 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7767692Z STAGE:2023-01-11 21:41:12 24647:24647 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7768261Z STAGE:2023-01-11 21:41:12 24646:24646 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:41:12 24647:24647 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7768283Z 2023-01-11T21:44:22.7768875Z STAGE:2023-01-11 21:41:12 24646:24646 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:41:12 24647:24647 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7768944Z 2023-01-11T21:44:22.7769295Z STAGE:2023-01-11 21:41:12 24647:24647 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7769636Z STAGE:2023-01-11 21:41:12 24646:24646 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7769982Z STAGE:2023-01-11 21:41:12 24646:24646 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7770504Z STAGE:2023-01-11 21:41:12 24647:24647 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7770890Z STAGE:2023-01-11 21:41:12 24646:24646 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7771253Z STAGE:2023-01-11 21:41:12 24647:24647 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7771340Z ok (4.264s) 2023-01-11T21:44:22.7771383Z 2023-01-11T21:44:22.7771647Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7771766Z Ran 1 test in 4.265s 2023-01-11T21:44:22.7771789Z 2023-01-11T21:44:22.7771876Z OK 2023-01-11T21:44:22.7771895Z 2023-01-11T21:44:22.7772024Z Generating XML reports... 2023-01-11T21:44:22.7772490Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214109.xml 2023-01-11T21:44:22.7772881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7773132Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7773547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7773728Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7773766Z 2023-01-11T21:44:22.7773898Z Running tests... 2023-01-11T21:44:22.7774179Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7774506Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7774758Z test_scatter (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7774986Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24759 2023-01-11T21:44:22.7775216Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24760 2023-01-11T21:44:22.7775608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7775771Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7776171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7776366Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7776751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7776937Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7777340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7777534Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7777788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7778047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7778446Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7778863Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7779098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7779412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7779766Z STAGE:2023-01-11 21:41:19 24759:24759 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7780108Z STAGE:2023-01-11 21:41:19 24760:24760 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7780462Z STAGE:2023-01-11 21:41:19 24759:24759 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7780809Z STAGE:2023-01-11 21:41:19 24760:24760 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7781173Z STAGE:2023-01-11 21:41:19 24759:24759 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7781522Z STAGE:2023-01-11 21:41:19 24760:24760 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7781862Z STAGE:2023-01-11 21:41:19 24759:24759 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7782204Z STAGE:2023-01-11 21:41:19 24760:24760 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7782554Z STAGE:2023-01-11 21:41:19 24760:24760 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7782903Z STAGE:2023-01-11 21:41:19 24759:24759 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7783311Z STAGE:2023-01-11 21:41:19 24760:24760 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7783684Z STAGE:2023-01-11 21:41:19 24759:24759 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7783786Z ok (4.217s) 2023-01-11T21:44:22.7783807Z 2023-01-11T21:44:22.7784087Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7784185Z Ran 1 test in 4.217s 2023-01-11T21:44:22.7784205Z 2023-01-11T21:44:22.7784296Z OK 2023-01-11T21:44:22.7784315Z 2023-01-11T21:44:22.7784442Z Generating XML reports... 2023-01-11T21:44:22.7784908Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214115.xml 2023-01-11T21:44:22.7785298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7785479Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7785882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7786079Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7786099Z 2023-01-11T21:44:22.7786192Z Running tests... 2023-01-11T21:44:22.7786476Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7786800Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7787059Z test_scatter_checks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7787287Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24872 2023-01-11T21:44:22.7787514Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24873 2023-01-11T21:44:22.7787901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7788082Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7788480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7788658Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7789042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7789220Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7789671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7789867Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7790117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7790370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7790791Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7791187Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7791426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7791663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7791770Z ok (4.171s) 2023-01-11T21:44:22.7791790Z 2023-01-11T21:44:22.7792077Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7792195Z Ran 1 test in 4.171s 2023-01-11T21:44:22.7792215Z 2023-01-11T21:44:22.7792308Z OK 2023-01-11T21:44:22.7792327Z 2023-01-11T21:44:22.7792454Z Generating XML reports... 2023-01-11T21:44:22.7792969Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214122.xml 2023-01-11T21:44:22.7793347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7793529Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7793929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7794127Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7794151Z 2023-01-11T21:44:22.7794262Z Running tests... 2023-01-11T21:44:22.7794545Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7794874Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7795137Z test_scatter_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7795349Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24981 2023-01-11T21:44:22.7795575Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24982 2023-01-11T21:44:22.7795963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7796141Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7796543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7796743Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7797130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7797308Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7797703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7797881Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7798129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7798374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7798789Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7799277Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7799516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7799754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7800106Z STAGE:2023-01-11 21:41:33 24981:24981 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7800446Z STAGE:2023-01-11 21:41:33 24982:24982 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7800781Z STAGE:2023-01-11 21:41:33 24982:24982 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7801126Z STAGE:2023-01-11 21:41:33 24981:24981 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7801492Z STAGE:2023-01-11 21:41:33 24982:24982 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7801861Z STAGE:2023-01-11 21:41:33 24981:24981 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7802205Z STAGE:2023-01-11 21:41:33 24982:24982 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7802547Z STAGE:2023-01-11 21:41:33 24981:24981 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7802941Z STAGE:2023-01-11 21:41:33 24982:24982 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7803297Z STAGE:2023-01-11 21:41:33 24981:24981 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7803663Z STAGE:2023-01-11 21:41:33 24982:24982 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7804008Z STAGE:2023-01-11 21:41:33 24981:24981 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7804112Z ok (4.246s) 2023-01-11T21:44:22.7804135Z 2023-01-11T21:44:22.7804416Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7804531Z Ran 1 test in 4.246s 2023-01-11T21:44:22.7804551Z 2023-01-11T21:44:22.7804645Z OK 2023-01-11T21:44:22.7804664Z 2023-01-11T21:44:22.7804785Z Generating XML reports... 2023-01-11T21:44:22.7805253Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214129.xml 2023-01-11T21:44:22.7805645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7805810Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7806211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7806405Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7806425Z 2023-01-11T21:44:22.7806534Z Running tests... 2023-01-11T21:44:22.7806817Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7807150Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7807417Z test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s) 2023-01-11T21:44:22.7807436Z 2023-01-11T21:44:22.7807774Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7807895Z Ran 1 test in 0.002s 2023-01-11T21:44:22.7807915Z 2023-01-11T21:44:22.7808007Z OK (skipped=1) 2023-01-11T21:44:22.7808026Z 2023-01-11T21:44:22.7808151Z Generating XML reports... 2023-01-11T21:44:22.7808623Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214136.xml 2023-01-11T21:44:22.7809012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7809255Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7809657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7809851Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7809871Z 2023-01-11T21:44:22.7809983Z Running tests... 2023-01-11T21:44:22.7810262Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7810849Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7811130Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s) 2023-01-11T21:44:22.7811152Z 2023-01-11T21:44:22.7811433Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7811548Z Ran 1 test in 0.002s 2023-01-11T21:44:22.7811567Z 2023-01-11T21:44:22.7811673Z OK (skipped=1) 2023-01-11T21:44:22.7811696Z 2023-01-11T21:44:22.7811821Z Generating XML reports... 2023-01-11T21:44:22.7812289Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214138.xml 2023-01-11T21:44:22.7812677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7812839Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7813328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7813533Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7813552Z 2023-01-11T21:44:22.7813662Z Running tests... 2023-01-11T21:44:22.7813944Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7814269Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7814540Z test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7814768Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25160 2023-01-11T21:44:22.7814992Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25161 2023-01-11T21:44:22.7815364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7815548Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7815947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7816142Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7816526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7816706Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7817106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7817303Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7817537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7817790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7818210Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7818620Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7818858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7819097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7819422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T21:44:22.7819672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T21:44:22.7820090Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.7820508Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.7820836Z STAGE:2023-01-11 21:41:45 25161:25161 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7821179Z STAGE:2023-01-11 21:41:45 25160:25160 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7821530Z STAGE:2023-01-11 21:41:45 25160:25160 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7821880Z STAGE:2023-01-11 21:41:45 25161:25161 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7822249Z STAGE:2023-01-11 21:41:45 25160:25160 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7822614Z STAGE:2023-01-11 21:41:45 25161:25161 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7823222Z STAGE:2023-01-11 21:41:45 25160:25160 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:41:45 25161:25161 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7823244Z 2023-01-11T21:44:22.7823603Z STAGE:2023-01-11 21:41:45 25161:25161 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7823953Z STAGE:2023-01-11 21:41:45 25160:25160 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7824301Z STAGE:2023-01-11 21:41:45 25161:25161 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7824673Z STAGE:2023-01-11 21:41:45 25160:25160 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7824775Z ok (4.272s) 2023-01-11T21:44:22.7824794Z 2023-01-11T21:44:22.7825074Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7825187Z Ran 1 test in 4.272s 2023-01-11T21:44:22.7825206Z 2023-01-11T21:44:22.7825300Z OK 2023-01-11T21:44:22.7825319Z 2023-01-11T21:44:22.7825449Z Generating XML reports... 2023-01-11T21:44:22.7825919Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214141.xml 2023-01-11T21:44:22.7826305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7826468Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7826866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7827066Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7827086Z 2023-01-11T21:44:22.7827197Z Running tests... 2023-01-11T21:44:22.7827478Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7827805Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7828069Z test_scatter_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7828300Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25279 2023-01-11T21:44:22.7828510Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25280 2023-01-11T21:44:22.7828896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7829076Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7829532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7829725Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7830107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7830288Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7830685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7830879Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7831114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7831362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7831781Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7832192Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7832428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7832761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7832934Z skip: Skipped due to small world size. (4.114s) 2023-01-11T21:44:22.7832954Z 2023-01-11T21:44:22.7833242Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7833358Z Ran 1 test in 4.114s 2023-01-11T21:44:22.7833378Z 2023-01-11T21:44:22.7833470Z OK (skipped=1) 2023-01-11T21:44:22.7833489Z 2023-01-11T21:44:22.7833616Z Generating XML reports... 2023-01-11T21:44:22.7834088Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214147.xml 2023-01-11T21:44:22.7834480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7834664Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7835069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7835265Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7835284Z 2023-01-11T21:44:22.7835393Z Running tests... 2023-01-11T21:44:22.7835657Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7835985Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7836255Z test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7836485Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25388 2023-01-11T21:44:22.7836710Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25389 2023-01-11T21:44:22.7837101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7837282Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7837682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7837878Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7838244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7838425Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7838823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7839074Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7839325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7839573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7839995Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7840409Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7840647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7840868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7840971Z ok (4.262s) 2023-01-11T21:44:22.7840995Z 2023-01-11T21:44:22.7841275Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7841387Z Ran 1 test in 4.263s 2023-01-11T21:44:22.7841406Z 2023-01-11T21:44:22.7841497Z OK 2023-01-11T21:44:22.7841516Z 2023-01-11T21:44:22.7841643Z Generating XML reports... 2023-01-11T21:44:22.7842111Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214154.xml 2023-01-11T21:44:22.7842545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7842716Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7843115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7843304Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7843323Z 2023-01-11T21:44:22.7843431Z Running tests... 2023-01-11T21:44:22.7843712Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7844039Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7844286Z test_send_recv (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7844516Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25497 2023-01-11T21:44:22.7844729Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25498 2023-01-11T21:44:22.7845123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7845300Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7845694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7845885Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7846271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7846451Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7846849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7847043Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7847281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7847536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7847948Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7848356Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7848647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7848884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7848988Z ok (4.231s) 2023-01-11T21:44:22.7849008Z 2023-01-11T21:44:22.7849293Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7849392Z Ran 1 test in 4.231s 2023-01-11T21:44:22.7849432Z 2023-01-11T21:44:22.7849508Z OK 2023-01-11T21:44:22.7849526Z 2023-01-11T21:44:22.7849653Z Generating XML reports... 2023-01-11T21:44:22.7850125Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214201.xml 2023-01-11T21:44:22.7850744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7850934Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7851343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7851541Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7851562Z 2023-01-11T21:44:22.7851675Z Running tests... 2023-01-11T21:44:22.7851937Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7852338Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7852619Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7852847Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25606 2023-01-11T21:44:22.7853069Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25607 2023-01-11T21:44:22.7853458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7853639Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7854034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7854230Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7854603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7854783Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7855178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7855366Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7855613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7855863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7856278Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7856690Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7856912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7857152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7857254Z ok (4.276s) 2023-01-11T21:44:22.7857274Z 2023-01-11T21:44:22.7857558Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7857674Z Ran 1 test in 4.276s 2023-01-11T21:44:22.7857694Z 2023-01-11T21:44:22.7857786Z OK 2023-01-11T21:44:22.7857805Z 2023-01-11T21:44:22.7857933Z Generating XML reports... 2023-01-11T21:44:22.7858482Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214208.xml 2023-01-11T21:44:22.7858868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7859033Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7859433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7859627Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7859647Z 2023-01-11T21:44:22.7859756Z Running tests... 2023-01-11T21:44:22.7860035Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7860363Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7860661Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7860893Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25715 2023-01-11T21:44:22.7861103Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25716 2023-01-11T21:44:22.7861492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7861673Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7862123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7862325Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7862713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7862889Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7863284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7863479Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7863714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7863965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7864383Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7864797Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7865035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7865274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7865628Z STAGE:2023-01-11 21:42:18 25716:25716 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7865968Z STAGE:2023-01-11 21:42:18 25715:25715 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7866314Z STAGE:2023-01-11 21:42:18 25715:25715 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7866649Z STAGE:2023-01-11 21:42:18 25716:25716 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7867017Z STAGE:2023-01-11 21:42:18 25715:25715 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7867384Z STAGE:2023-01-11 21:42:18 25716:25716 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7867479Z ok (4.269s) 2023-01-11T21:44:22.7867498Z 2023-01-11T21:44:22.7867775Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7867884Z Ran 1 test in 4.269s 2023-01-11T21:44:22.7867967Z 2023-01-11T21:44:22.7868065Z OK 2023-01-11T21:44:22.7868084Z 2023-01-11T21:44:22.7868209Z Generating XML reports... 2023-01-11T21:44:22.7868668Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214214.xml 2023-01-11T21:44:22.7869054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7869238Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7869639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7869837Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7869857Z 2023-01-11T21:44:22.7869969Z Running tests... 2023-01-11T21:44:22.7870252Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7870583Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7870880Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7871089Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25828 2023-01-11T21:44:22.7871314Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25829 2023-01-11T21:44:22.7871750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7871937Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7872338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7872532Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7872919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7873103Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7873482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7873670Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7873957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7874209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7874621Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7875030Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7875262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7875499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7875845Z STAGE:2023-01-11 21:42:25 25828:25828 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7876171Z STAGE:2023-01-11 21:42:25 25829:25829 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7876517Z STAGE:2023-01-11 21:42:25 25828:25828 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7876880Z STAGE:2023-01-11 21:42:25 25828:25828 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7877227Z STAGE:2023-01-11 21:42:25 25829:25829 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7877588Z STAGE:2023-01-11 21:42:25 25829:25829 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7877689Z ok (4.262s) 2023-01-11T21:44:22.7877762Z 2023-01-11T21:44:22.7878050Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7878164Z Ran 1 test in 4.262s 2023-01-11T21:44:22.7878184Z 2023-01-11T21:44:22.7878277Z OK 2023-01-11T21:44:22.7878296Z 2023-01-11T21:44:22.7878405Z Generating XML reports... 2023-01-11T21:44:22.7878877Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214221.xml 2023-01-11T21:44:22.7879266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7879443Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7879837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7880031Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7880051Z 2023-01-11T21:44:22.7880158Z Running tests... 2023-01-11T21:44:22.7880436Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7880747Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7881030Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7881254Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25941 2023-01-11T21:44:22.7881524Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25942 2023-01-11T21:44:22.7881913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7882092Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7882486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7882676Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7883065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7883227Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7883626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7883824Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7884075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7884325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7884738Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7885151Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7885393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7885629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7885960Z STAGE:2023-01-11 21:42:32 25942:25942 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7886308Z STAGE:2023-01-11 21:42:32 25941:25941 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7886661Z STAGE:2023-01-11 21:42:32 25942:25942 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7887025Z STAGE:2023-01-11 21:42:32 25942:25942 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7887372Z STAGE:2023-01-11 21:42:32 25941:25941 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7887736Z STAGE:2023-01-11 21:42:32 25941:25941 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7887891Z ok (4.219s) 2023-01-11T21:44:22.7887910Z 2023-01-11T21:44:22.7888189Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7888286Z Ran 1 test in 4.220s 2023-01-11T21:44:22.7888319Z 2023-01-11T21:44:22.7888394Z OK 2023-01-11T21:44:22.7888413Z 2023-01-11T21:44:22.7888533Z Generating XML reports... 2023-01-11T21:44:22.7889007Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214228.xml 2023-01-11T21:44:22.7889395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7889578Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7889975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7890175Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7890194Z 2023-01-11T21:44:22.7890470Z Running tests... 2023-01-11T21:44:22.7890751Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7891081Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7891324Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s) 2023-01-11T21:44:22.7891345Z 2023-01-11T21:44:22.7891695Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7891813Z Ran 1 test in 0.002s 2023-01-11T21:44:22.7891833Z 2023-01-11T21:44:22.7891942Z OK (skipped=1) 2023-01-11T21:44:22.7891961Z 2023-01-11T21:44:22.7892082Z Generating XML reports... 2023-01-11T21:44:22.7892558Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214235.xml 2023-01-11T21:44:22.7892946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7893114Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7893511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7893707Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7893726Z 2023-01-11T21:44:22.7893844Z Running tests... 2023-01-11T21:44:22.7894125Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7894456Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7894732Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s) 2023-01-11T21:44:22.7894752Z 2023-01-11T21:44:22.7895032Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7895133Z Ran 1 test in 0.002s 2023-01-11T21:44:22.7895170Z 2023-01-11T21:44:22.7895264Z OK (skipped=1) 2023-01-11T21:44:22.7895283Z 2023-01-11T21:44:22.7895407Z Generating XML reports... 2023-01-11T21:44:22.7895875Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214237.xml 2023-01-11T21:44:22.7896263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7896449Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7896850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7897044Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7897064Z 2023-01-11T21:44:22.7897172Z Running tests... 2023-01-11T21:44:22.7897435Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7897851Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7898122Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s) 2023-01-11T21:44:22.7898142Z 2023-01-11T21:44:22.7898420Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7898534Z Ran 1 test in 0.002s 2023-01-11T21:44:22.7898553Z 2023-01-11T21:44:22.7898662Z OK (skipped=1) 2023-01-11T21:44:22.7898685Z 2023-01-11T21:44:22.7898812Z Generating XML reports... 2023-01-11T21:44:22.7899278Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214240.xml 2023-01-11T21:44:22.7899669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7899831Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7900238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7900432Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7900451Z 2023-01-11T21:44:22.7900562Z Running tests... 2023-01-11T21:44:22.7900844Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7901170Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7901495Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7901730Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26153 2023-01-11T21:44:22.7901940Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26154 2023-01-11T21:44:22.7902332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7902516Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7902912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7903103Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7903484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7903667Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7904056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7904250Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7904485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7904737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7905154Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7905567Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7905802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7906043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7906387Z STAGE:2023-01-11 21:42:46 26153:26153 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7906727Z STAGE:2023-01-11 21:42:46 26154:26154 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7907076Z STAGE:2023-01-11 21:42:46 26154:26154 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7907484Z STAGE:2023-01-11 21:42:46 26154:26154 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7907836Z STAGE:2023-01-11 21:42:46 26153:26153 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7908200Z STAGE:2023-01-11 21:42:46 26153:26153 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7908304Z ok (4.237s) 2023-01-11T21:44:22.7908324Z 2023-01-11T21:44:22.7908606Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7908719Z Ran 1 test in 4.237s 2023-01-11T21:44:22.7908739Z 2023-01-11T21:44:22.7908832Z OK 2023-01-11T21:44:22.7908850Z 2023-01-11T21:44:22.7908980Z Generating XML reports... 2023-01-11T21:44:22.7909429Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214242.xml 2023-01-11T21:44:22.7909813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7909996Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7910395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7910595Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7910615Z 2023-01-11T21:44:22.7910726Z Running tests... 2023-01-11T21:44:22.7911061Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7911395Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7911660Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7911871Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26266 2023-01-11T21:44:22.7912098Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26267 2023-01-11T21:44:22.7912490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7912670Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7913066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7913263Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7913653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7913834Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7914213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7914408Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7914660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7914916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7915331Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7915745Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7915985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7916224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7916327Z ok (4.224s) 2023-01-11T21:44:22.7916348Z 2023-01-11T21:44:22.7916610Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7916730Z Ran 1 test in 4.224s 2023-01-11T21:44:22.7916749Z 2023-01-11T21:44:22.7916899Z OK 2023-01-11T21:44:22.7916918Z 2023-01-11T21:44:22.7917048Z Generating XML reports... 2023-01-11T21:44:22.7917522Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214249.xml 2023-01-11T21:44:22.7917911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7918092Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7918492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7918690Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7918710Z 2023-01-11T21:44:22.7918802Z Running tests... 2023-01-11T21:44:22.7919081Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7919410Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7919706Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7919932Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26375 2023-01-11T21:44:22.7920160Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26376 2023-01-11T21:44:22.7920549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7920774Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7921162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7921357Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7921745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7921928Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7922323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7922513Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7922762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7923012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7923428Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7923824Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7924062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7924303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7924648Z STAGE:2023-01-11 21:42:59 26375:26375 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7924991Z STAGE:2023-01-11 21:42:59 26376:26376 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7925342Z STAGE:2023-01-11 21:42:59 26375:26375 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7925708Z STAGE:2023-01-11 21:42:59 26375:26375 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7926056Z STAGE:2023-01-11 21:42:59 26376:26376 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7926419Z STAGE:2023-01-11 21:42:59 26376:26376 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7926506Z ok (4.221s) 2023-01-11T21:44:22.7926525Z 2023-01-11T21:44:22.7926809Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7926978Z Ran 1 test in 4.221s 2023-01-11T21:44:22.7926998Z 2023-01-11T21:44:22.7927090Z OK 2023-01-11T21:44:22.7927110Z 2023-01-11T21:44:22.7927236Z Generating XML reports... 2023-01-11T21:44:22.7927711Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214256.xml 2023-01-11T21:44:22.7928100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7928278Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7928658Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7928850Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7928870Z 2023-01-11T21:44:22.7928980Z Running tests... 2023-01-11T21:44:22.7929254Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7929589Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7929876Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7930100Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26488 2023-01-11T21:44:22.7930496Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26489 2023-01-11T21:44:22.7930972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7931143Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7931545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7931735Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7932126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7932304Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7932698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7932888Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7933138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7933373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7933783Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7934189Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7934425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7934659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7935007Z STAGE:2023-01-11 21:43:06 26489:26489 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7935348Z STAGE:2023-01-11 21:43:06 26488:26488 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T21:44:22.7935696Z STAGE:2023-01-11 21:43:06 26488:26488 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7936059Z STAGE:2023-01-11 21:43:06 26488:26488 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7936390Z STAGE:2023-01-11 21:43:06 26489:26489 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T21:44:22.7936754Z STAGE:2023-01-11 21:43:06 26489:26489 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T21:44:22.7936941Z ok (4.205s) 2023-01-11T21:44:22.7936962Z 2023-01-11T21:44:22.7937240Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7937352Z Ran 1 test in 4.206s 2023-01-11T21:44:22.7937371Z 2023-01-11T21:44:22.7937461Z OK 2023-01-11T21:44:22.7937480Z 2023-01-11T21:44:22.7937599Z Generating XML reports... 2023-01-11T21:44:22.7938067Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214302.xml 2023-01-11T21:44:22.7938452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7938616Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7939012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7939203Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7939226Z 2023-01-11T21:44:22.7939332Z Running tests... 2023-01-11T21:44:22.7939606Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7939934Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7940200Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7940474Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26601 2023-01-11T21:44:22.7940690Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26602 2023-01-11T21:44:22.7941080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7941259Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7941657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7941856Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7942238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7942413Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7942809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7943002Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7943238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7943486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7943903Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7944319Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7944551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7944789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7944888Z ok (4.259s) 2023-01-11T21:44:22.7944908Z 2023-01-11T21:44:22.7945186Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7945283Z Ran 1 test in 4.260s 2023-01-11T21:44:22.7945315Z 2023-01-11T21:44:22.7945388Z OK 2023-01-11T21:44:22.7945407Z 2023-01-11T21:44:22.7945530Z Generating XML reports... 2023-01-11T21:44:22.7946003Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214309.xml 2023-01-11T21:44:22.7946392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7946629Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7947031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7947228Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7947247Z 2023-01-11T21:44:22.7947358Z Running tests... 2023-01-11T21:44:22.7947625Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7947952Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7948226Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7948451Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26800 2023-01-11T21:44:22.7948676Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26801 2023-01-11T21:44:22.7949076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7949245Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7949640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7949837Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7950253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7950438Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7950828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7951013Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7951254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7951497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7951902Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7952306Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7952525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7952753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7952845Z ok (5.886s) 2023-01-11T21:44:22.7952865Z 2023-01-11T21:44:22.7953144Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7953258Z Ran 1 test in 5.886s 2023-01-11T21:44:22.7953278Z 2023-01-11T21:44:22.7953374Z OK 2023-01-11T21:44:22.7953393Z 2023-01-11T21:44:22.7953519Z Generating XML reports... 2023-01-11T21:44:22.7953987Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214316.xml 2023-01-11T21:44:22.7954375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7954540Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7954944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7955140Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7955160Z 2023-01-11T21:44:22.7955266Z Running tests... 2023-01-11T21:44:22.7955547Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7955876Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7956202Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7956427Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27001 2023-01-11T21:44:22.7956638Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27002 2023-01-11T21:44:22.7957027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7957207Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7957600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7957794Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7958179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7958363Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7958760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7958954Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7959190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7959483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7959905Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7960313Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7960546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7960789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7960888Z ok (6.272s) 2023-01-11T21:44:22.7960908Z 2023-01-11T21:44:22.7961186Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7961284Z Ran 1 test in 6.273s 2023-01-11T21:44:22.7961317Z 2023-01-11T21:44:22.7961392Z OK 2023-01-11T21:44:22.7961411Z 2023-01-11T21:44:22.7961531Z Generating XML reports... 2023-01-11T21:44:22.7962003Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214324.xml 2023-01-11T21:44:22.7962388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7962568Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7962964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7963169Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7963189Z 2023-01-11T21:44:22.7963299Z Running tests... 2023-01-11T21:44:22.7963564Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7963892Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7964162Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7964387Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27116 2023-01-11T21:44:22.7964613Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27117 2023-01-11T21:44:22.7964999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7965183Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7965640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7965817Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7966202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7966384Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7966786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7966985Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7967240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7967494Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7967908Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7968324Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7968542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7968778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7968884Z ok (4.253s) 2023-01-11T21:44:22.7968971Z 2023-01-11T21:44:22.7969261Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7969376Z Ran 1 test in 4.254s 2023-01-11T21:44:22.7969395Z 2023-01-11T21:44:22.7969487Z OK 2023-01-11T21:44:22.7969506Z 2023-01-11T21:44:22.7969633Z Generating XML reports... 2023-01-11T21:44:22.7970103Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214333.xml 2023-01-11T21:44:22.7970666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7970839Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7971243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7971437Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7971458Z 2023-01-11T21:44:22.7971571Z Running tests... 2023-01-11T21:44:22.7971852Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7972181Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7972439Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7972667Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27229 2023-01-11T21:44:22.7972881Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27230 2023-01-11T21:44:22.7973267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7973445Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7973846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7974079Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7974469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7974648Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7975044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7975240Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7975561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7975814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7976231Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7976651Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7976888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7977121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7977219Z ok (5.755s) 2023-01-11T21:44:22.7977239Z 2023-01-11T21:44:22.7977521Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7977623Z Ran 1 test in 5.755s 2023-01-11T21:44:22.7977662Z 2023-01-11T21:44:22.7977737Z OK 2023-01-11T21:44:22.7977756Z 2023-01-11T21:44:22.7977883Z Generating XML reports... 2023-01-11T21:44:22.7978354Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214340.xml 2023-01-11T21:44:22.7978741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7978984Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7979394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7979587Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7979607Z 2023-01-11T21:44:22.7979718Z Running tests... 2023-01-11T21:44:22.7979980Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7980315Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7980618Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7980845Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27340 2023-01-11T21:44:22.7981069Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27341 2023-01-11T21:44:22.7981461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7981642Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7982035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7982213Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7982593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7982773Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7983168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7983366Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7983617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7983866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7984281Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7984694Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7985029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7985263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7986099Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-01-11T21:44:22.7986930Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-01-11T21:44:22.7987039Z ok (6.236s) 2023-01-11T21:44:22.7987059Z 2023-01-11T21:44:22.7987345Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7987505Z Ran 1 test in 6.237s 2023-01-11T21:44:22.7987525Z 2023-01-11T21:44:22.7987616Z OK 2023-01-11T21:44:22.7987635Z 2023-01-11T21:44:22.7987761Z Generating XML reports... 2023-01-11T21:44:22.7988230Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214348.xml 2023-01-11T21:44:22.7988618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7988800Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7989187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7989381Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7989400Z 2023-01-11T21:44:22.7989509Z Running tests... 2023-01-11T21:44:22.7989788Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7990121Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.7990408Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.7990633Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27455 2023-01-11T21:44:22.7990860Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27456 2023-01-11T21:44:22.7991250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7991417Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7991811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7992009Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7992402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7992584Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7992982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.7993178Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.7993431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.7993727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.7994147Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7994564Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.7994805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.7995040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.7995286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T21:44:22.7995535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T21:44:22.7995947Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.7996360Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.7996605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2023-01-11T21:44:22.7996837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2023-01-11T21:44:22.7997299Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2023-01-11T21:44:22.7997712Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2023-01-11T21:44:22.7997818Z ok (10.758s) 2023-01-11T21:44:22.7997838Z 2023-01-11T21:44:22.7998120Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.7998236Z Ran 1 test in 10.759s 2023-01-11T21:44:22.7998259Z 2023-01-11T21:44:22.7998351Z OK 2023-01-11T21:44:22.7998370Z 2023-01-11T21:44:22.7998493Z Generating XML reports... 2023-01-11T21:44:22.7998947Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214357.xml 2023-01-11T21:44:22.7999334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.7999520Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.7999916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.8000112Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.8000132Z 2023-01-11T21:44:22.8000241Z Running tests... 2023-01-11T21:44:22.8000522Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.8000850Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T21:44:22.8001147Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T21:44:22.8001358Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27578 2023-01-11T21:44:22.8001582Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27579 2023-01-11T21:44:22.8001966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.8002139Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.8002530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.8002724Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.8003109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T21:44:22.8003353Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T21:44:22.8003737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T21:44:22.8003927Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T21:44:22.8004177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T21:44:22.8004430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T21:44:22.8004840Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.8005250Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T21:44:22.8005483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T21:44:22.8005724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T21:44:22.8005975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T21:44:22.8006208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T21:44:22.8006669Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.8007089Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T21:44:22.8007335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2023-01-11T21:44:22.8007579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2023-01-11T21:44:22.8007986Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2023-01-11T21:44:22.8008399Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2023-01-11T21:44:22.8008502Z ok (10.873s) 2023-01-11T21:44:22.8008521Z 2023-01-11T21:44:22.8008799Z ---------------------------------------------------------------------- 2023-01-11T21:44:22.8008898Z Ran 1 test in 10.873s 2023-01-11T21:44:22.8008932Z 2023-01-11T21:44:22.8009010Z OK 2023-01-11T21:44:22.8009029Z 2023-01-11T21:44:22.8009154Z Generating XML reports... 2023-01-11T21:44:22.8009621Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214410.xml 2023-01-11T21:44:22.8009640Z 2023-01-11T21:44:22.8010073Z ##[endgroup] 2023-01-11T21:44:22.8010817Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_amx6tdux) 2023-01-11T21:44:22.8010847Z 2023-01-11T21:44:22.8011073Z Running distributed tests for the gloo backend with file init_method in shard 1 of 3 2023-01-11T21:44:22.8011612Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:44:22.549022] 2023-01-11T22:14:04.7648388Z 2023-01-11T22:14:04.7649222Z Expand the folded group to see the log file of distributed/test_distributed_spawn 2023-01-11T22:14:04.7651312Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_rj7_8efa) 2023-01-11T22:14:04.7653200Z 2023-01-11T22:14:04.7704701Z , <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_cat_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_stack_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_apply_optim_in_backward>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_apply_optim_in_backward_grad_as_bucket_view_false>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_apply_optim_in_backward_ignored_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_zero_output_features>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]> 2023-01-11T22:14:04.7757943Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7758668Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7762077Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7762862Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7763620Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7764427Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7765308Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7766170Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7767027Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7767907Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7769069Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7770074Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7771248Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7772011Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7772486Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7772971Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7773597Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7774386Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7775207Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7776042Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7776927Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7777652Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7778420Z test_all_gather (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7779129Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7779975Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7780767Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7781562Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7782326Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7783100Z test_all_gather_complex (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7783826Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7784485Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7784895Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7785334Z test_all_gather_group (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7786074Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7786864Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7787633Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7788282Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7788989Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7789749Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7790330Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7791036Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7791830Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7792295Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7792946Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7793627Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7794309Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7795112Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7795780Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7796167Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7796712Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7797149Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7797532Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7798129Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7798554Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7798981Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7799367Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7800107Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7800718Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7801429Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7802171Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7802865Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7803455Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7803890Z test_all_reduce_max (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7804457Z test_all_reduce_min (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7805168Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7805711Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7806124Z test_all_reduce_product (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7806511Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7807112Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7807496Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7807890Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7808362Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7809021Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7809436Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7809816Z test_all_to_all (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7810169Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7810908Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7811296Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7811674Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7812396Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7812806Z test_all_to_all_group (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7813182Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7813881Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7814689Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7815494Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7816204Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7816917Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7817609Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7818064Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7818482Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7819125Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7819954Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7820802Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7821590Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7822358Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7823018Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7823865Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7824609Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7825252Z test_average_parameters (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7825626Z test_backend_full_group (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7826008Z test_backend_group (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7826373Z test_barrier (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7826740Z test_barrier_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7827103Z test_barrier_full_group (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7827490Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7827878Z test_barrier_group (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7828238Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7828721Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7829141Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7829511Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7829905Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7830306Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7830731Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7831130Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7831539Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7831953Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7832338Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7832768Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7833192Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7833602Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7833972Z test_broadcast (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7834342Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7834727Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7835099Z test_broadcast_group (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7835484Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7835872Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7836299Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7836803Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7837263Z test_ddp_apply_optim_in_backward (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7837718Z test_ddp_apply_optim_in_backward_grad_as_bucket_view_false (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7838170Z test_ddp_apply_optim_in_backward_ignored_params (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7838592Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7838994Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7839452Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7839878Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7840322Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7840781Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7841439Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7841870Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7842301Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7842686Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7843052Z test_ddp_device (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7843434Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7843846Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7844472Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7844942Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7845368Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7845759Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7846228Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7846928Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7847405Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7847963Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7848568Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7849159Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7849742Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7850736Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7851338Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7851928Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7852524Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7853044Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7853526Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7853961Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7854324Z test_ddp_inference (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7854715Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7855114Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7855499Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7855892Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7856321Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7856872Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7857313Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7857723Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7858110Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7858503Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7858935Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7859355Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7859761Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7860145Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7860560Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7860986Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7861381Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7861776Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7862177Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7862657Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7863049Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7863460Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7863907Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7864317Z test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7864713Z test_destroy_full_group (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7865094Z test_destroy_group (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7865468Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7865877Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7866286Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7866659Z test_gather (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7867003Z test_gather_checks (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7867364Z test_gather_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7867733Z test_gather_full_group (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7868085Z test_gather_group (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7868446Z test_gather_object (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7868821Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7869180Z test_get_backend (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7869539Z test_get_future (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7869891Z test_get_rank (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7870261Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7870631Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7871012Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7871377Z test_irecv (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7871699Z test_isend (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7872064Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7872462Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7872848Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7873293Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7873801Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7874209Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7874609Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7875034Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7875454Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7875857Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7876263Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7876667Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7877047Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7877448Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7877835Z test_new_subgroups (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7878227Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7878664Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7879141Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7879600Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7880071Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7880535Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7880980Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7881403Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7881797Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7882221Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7882649Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7883069Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7883540Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7884043Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7884518Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7884910Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7885298Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7885692Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7886073Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7886453Z test_reduce_group_max (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7886823Z test_reduce_group_min (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7887206Z test_reduce_group_product (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7887567Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7887928Z test_reduce_max (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7888285Z test_reduce_min (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7888636Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7889005Z test_reduce_product (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7889390Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7889769Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7890137Z test_reduce_sum (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7891146Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7891510Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7891892Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7892252Z test_scatter (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7892608Z test_scatter_checks (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7892964Z test_scatter_complex (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7893336Z test_scatter_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7893712Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7894078Z test_scatter_full_group (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7894452Z test_scatter_group (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7894829Z test_scatter_object_list (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7895174Z test_send_recv (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7895547Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7895961Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7896400Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7896803Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7897191Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7897661Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7898078Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7898486Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7898873Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7899260Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7899686Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7900095Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7900493Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7900877Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7901266Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7901641Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7902035Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7902471Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7902905Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.7903632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.7904069Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.7904640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.7905098Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.7905323Z 2023-01-11T22:14:04.7905416Z Running tests... 2023-01-11T22:14:04.7905823Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.7906351Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.7906928Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.7907464Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27734 2023-01-11T22:14:04.7907901Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27735 2023-01-11T22:14:04.7908609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.7909054Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.7909603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.7910060Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.7910629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.7911044Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.7911605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.7912059Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.7912502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.7912980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.7913627Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.7914305Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.7914879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.7915334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.7915847Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2023-01-11T22:14:04.7916676Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2023-01-11T22:14:04.7917317Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2023-01-11T22:14:04.7918127Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2023-01-11T22:14:04.7918773Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2023-01-11T22:14:04.7919326Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2023-01-11T22:14:04.7920115Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2023-01-11T22:14:04.7921004Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2023-01-11T22:14:04.7921656Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2023-01-11T22:14:04.7922210Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2023-01-11T22:14:04.7923017Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2023-01-11T22:14:04.7923880Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2023-01-11T22:14:04.7924530Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2023-01-11T22:14:04.7925085Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2023-01-11T22:14:04.7925959Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2023-01-11T22:14:04.7926833Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2023-01-11T22:14:04.7927309Z ok (6.884s) 2023-01-11T22:14:04.7927458Z 2023-01-11T22:14:04.7927729Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.7928036Z Ran 1 test in 6.884s 2023-01-11T22:14:04.7928194Z 2023-01-11T22:14:04.7928285Z OK 2023-01-11T22:14:04.7928417Z 2023-01-11T22:14:04.7928540Z Generating XML reports... 2023-01-11T22:14:04.7929142Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214426.xml 2023-01-11T22:14:04.7929830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.7930621Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.7931232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.7931702Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.7931910Z 2023-01-11T22:14:04.7932019Z Running tests... 2023-01-11T22:14:04.7932504Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.7933053Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.7933561Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.003s) 2023-01-11T22:14:04.7933865Z 2023-01-11T22:14:04.7934131Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.7934457Z Ran 1 test in 0.004s 2023-01-11T22:14:04.7934615Z 2023-01-11T22:14:04.7934723Z OK (skipped=1) 2023-01-11T22:14:04.7934858Z 2023-01-11T22:14:04.7934979Z Generating XML reports... 2023-01-11T22:14:04.7935575Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214436.xml 2023-01-11T22:14:04.7936277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.7936708Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.7937270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.7937728Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.7938181Z 2023-01-11T22:14:04.7938298Z Running tests... 2023-01-11T22:14:04.7938690Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.7939226Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.7939734Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.7940199Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27879 2023-01-11T22:14:04.7940650Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27880 2023-01-11T22:14:04.7941253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.7941699Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.7942249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.7942717Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.7943292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.7943817Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.7944413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.7944880Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.7945331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.7945802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.7946458Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.7947132Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.7947644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.7948092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.7948425Z ok (4.249s) 2023-01-11T22:14:04.7948571Z 2023-01-11T22:14:04.7948870Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.7949208Z Ran 1 test in 4.249s 2023-01-11T22:14:04.7949350Z 2023-01-11T22:14:04.7949443Z OK 2023-01-11T22:14:04.7949580Z 2023-01-11T22:14:04.7949771Z Generating XML reports... 2023-01-11T22:14:04.7950380Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214438.xml 2023-01-11T22:14:04.7951069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.7951521Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.7952094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.7952567Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.7952773Z 2023-01-11T22:14:04.7952928Z Running tests... 2023-01-11T22:14:04.7953333Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.7953858Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.7954379Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.7955445Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.658s) 2023-01-11T22:14:04.7955982Z 2023-01-11T22:14:04.7956227Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.7956552Z Ran 1 test in 1.658s 2023-01-11T22:14:04.7956710Z 2023-01-11T22:14:04.7956816Z OK (skipped=1) 2023-01-11T22:14:04.7956969Z 2023-01-11T22:14:04.7957075Z Generating XML reports... 2023-01-11T22:14:04.7957673Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214445.xml 2023-01-11T22:14:04.7958374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.7958819Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.7959368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.7959825Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.7960050Z 2023-01-11T22:14:04.7960156Z Running tests... 2023-01-11T22:14:04.7960552Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.7961125Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.7961657Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.7962165Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28022 2023-01-11T22:14:04.7962588Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28023 2023-01-11T22:14:04.7963182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.7963624Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.7964210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.7964659Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.7965230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.7965672Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.7966255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.7966705Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.7967217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.7967711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.7968368Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.7969029Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.7969544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.7970005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.7970662Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.7971128Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.7971482Z ok (4.215s) 2023-01-11T22:14:04.7971629Z 2023-01-11T22:14:04.7971901Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.7972208Z Ran 1 test in 4.215s 2023-01-11T22:14:04.7972367Z 2023-01-11T22:14:04.7972459Z OK 2023-01-11T22:14:04.7972591Z 2023-01-11T22:14:04.7972715Z Generating XML reports... 2023-01-11T22:14:04.7973297Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214449.xml 2023-01-11T22:14:04.7974006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.7974450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.7975015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.7975477Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.7975730Z 2023-01-11T22:14:04.7975850Z Running tests... 2023-01-11T22:14:04.7976266Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.7976803Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.7977355Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.7977865Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28135 2023-01-11T22:14:04.7978401Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28136 2023-01-11T22:14:04.7979010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.7979436Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.7980008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.7980476Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.7981049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.7981469Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.7982036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.7982503Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.7982951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.7983447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.7984131Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.7984908Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.7985435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.7985882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.7986354Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.7986850Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.7987210Z ok (4.237s) 2023-01-11T22:14:04.7987337Z 2023-01-11T22:14:04.7987611Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.7987940Z Ran 1 test in 4.237s 2023-01-11T22:14:04.7988101Z 2023-01-11T22:14:04.7988198Z OK 2023-01-11T22:14:04.7988333Z 2023-01-11T22:14:04.7988439Z Generating XML reports... 2023-01-11T22:14:04.7989050Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214456.xml 2023-01-11T22:14:04.7989756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.7990197Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.7990747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.7991212Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.7991436Z 2023-01-11T22:14:04.7991543Z Running tests... 2023-01-11T22:14:04.7991923Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.7992443Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.7992981Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.7993501Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28248 2023-01-11T22:14:04.7993927Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28249 2023-01-11T22:14:04.7994515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.7994959Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.7995509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.7996036Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.7996607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.7997044Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.7997589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.7998046Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.7998491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.7998976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.7999609Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8000290Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8000800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8001242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8001761Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.8002250Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.8002720Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.8003170Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.8003632Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.8004100Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.8004544Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.8005010Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.8005470Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.8005933Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.8006377Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.8006832Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.8007168Z ok (6.924s) 2023-01-11T22:14:04.8007315Z 2023-01-11T22:14:04.8007590Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8007903Z Ran 1 test in 6.924s 2023-01-11T22:14:04.8008060Z 2023-01-11T22:14:04.8008151Z OK 2023-01-11T22:14:04.8008285Z 2023-01-11T22:14:04.8008407Z Generating XML reports... 2023-01-11T22:14:04.8008995Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214503.xml 2023-01-11T22:14:04.8009705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8010153Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8011006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8011454Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8011678Z 2023-01-11T22:14:04.8011788Z Running tests... 2023-01-11T22:14:04.8012185Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8012786Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8013342Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8013868Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28363 2023-01-11T22:14:04.8014305Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28364 2023-01-11T22:14:04.8014891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8015332Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8015899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8016346Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8016915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8017355Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8017918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8018353Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8018864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8019366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8020018Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8020678Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8021191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8021655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8022106Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.8022582Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.8023056Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.8023523Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.8023847Z ok (5.743s) 2023-01-11T22:14:04.8023992Z 2023-01-11T22:14:04.8024263Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8024590Z Ran 1 test in 5.743s 2023-01-11T22:14:04.8024749Z 2023-01-11T22:14:04.8024824Z OK 2023-01-11T22:14:04.8024956Z 2023-01-11T22:14:04.8025082Z Generating XML reports... 2023-01-11T22:14:04.8025682Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214512.xml 2023-01-11T22:14:04.8026384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8026812Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8027380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8027844Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8028070Z 2023-01-11T22:14:04.8028165Z Running tests... 2023-01-11T22:14:04.8028565Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8029087Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8029647Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8030245Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28478 2023-01-11T22:14:04.8030681Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28479 2023-01-11T22:14:04.8031285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8031708Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8032276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8032731Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8033300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8033720Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8034287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8034742Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8035185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8035707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8036367Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8037041Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8037530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8037991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8038462Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.8038935Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.8039395Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.8039868Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.8040334Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.8040797Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.8041250Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.8041712Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.8042053Z ok (5.845s) 2023-01-11T22:14:04.8042198Z 2023-01-11T22:14:04.8042454Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8042776Z Ran 1 test in 5.846s 2023-01-11T22:14:04.8042936Z 2023-01-11T22:14:04.8043031Z OK 2023-01-11T22:14:04.8043162Z 2023-01-11T22:14:04.8043268Z Generating XML reports... 2023-01-11T22:14:04.8043869Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214520.xml 2023-01-11T22:14:04.8044630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8045077Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8045628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8046089Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8046412Z 2023-01-11T22:14:04.8046520Z Running tests... 2023-01-11T22:14:04.8046919Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8047425Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8048007Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8048566Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28593 2023-01-11T22:14:04.8048991Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28594 2023-01-11T22:14:04.8049587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8050028Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8050799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8051251Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8051823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8052261Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8052880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8053350Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8053793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8054281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8054918Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8055602Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8056112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8056575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8057024Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.8057505Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.8057849Z ok (6.066s) 2023-01-11T22:14:04.8057993Z 2023-01-11T22:14:04.8058243Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8058564Z Ran 1 test in 6.066s 2023-01-11T22:14:04.8058721Z 2023-01-11T22:14:04.8058813Z OK 2023-01-11T22:14:04.8058944Z 2023-01-11T22:14:04.8059065Z Generating XML reports... 2023-01-11T22:14:04.8059654Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214529.xml 2023-01-11T22:14:04.8060359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8060800Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8061351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8061817Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8062041Z 2023-01-11T22:14:04.8062148Z Running tests... 2023-01-11T22:14:04.8062546Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8063045Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8063621Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8064257Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28708 2023-01-11T22:14:04.8064675Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28709 2023-01-11T22:14:04.8065276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8065728Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8066293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8066738Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8067307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8067741Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8068307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8068743Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8069187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8069674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8070361Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8071053Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8071560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8072018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8072474Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.8072950Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.8073420Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.8073885Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.8074215Z ok (6.664s) 2023-01-11T22:14:04.8074363Z 2023-01-11T22:14:04.8074633Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8074955Z Ran 1 test in 6.664s 2023-01-11T22:14:04.8075112Z 2023-01-11T22:14:04.8075188Z OK 2023-01-11T22:14:04.8075319Z 2023-01-11T22:14:04.8075439Z Generating XML reports... 2023-01-11T22:14:04.8076037Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214537.xml 2023-01-11T22:14:04.8076746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8077172Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8077733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8078191Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8078418Z 2023-01-11T22:14:04.8078512Z Running tests... 2023-01-11T22:14:04.8078912Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8079433Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8079986Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8080497Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28823 2023-01-11T22:14:04.8081000Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28824 2023-01-11T22:14:04.8081599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8082022Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8082588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8083047Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8083622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8084043Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8084602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8085061Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8085489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8085975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8086619Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8087350Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8087855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8088319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8088778Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.8089257Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.8089708Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.8090176Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.8090781Z ok (6.253s) 2023-01-11T22:14:04.8090930Z 2023-01-11T22:14:04.8091187Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8091515Z Ran 1 test in 6.254s 2023-01-11T22:14:04.8091676Z 2023-01-11T22:14:04.8091768Z OK 2023-01-11T22:14:04.8091901Z 2023-01-11T22:14:04.8092023Z Generating XML reports... 2023-01-11T22:14:04.8092616Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214547.xml 2023-01-11T22:14:04.8093318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8093767Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8094314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8094778Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8095003Z 2023-01-11T22:14:04.8095110Z Running tests... 2023-01-11T22:14:04.8095510Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8096019Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8096596Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8097148Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28938 2023-01-11T22:14:04.8097567Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28939 2023-01-11T22:14:04.8098265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8098708Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8099276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8099720Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8100294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8100734Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8101295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8101735Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8102180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8102670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8103297Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8103978Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8104556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8105032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8105484Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.8105961Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.8106429Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.8106903Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.8107231Z ok (5.712s) 2023-01-11T22:14:04.8107375Z 2023-01-11T22:14:04.8107647Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8107972Z Ran 1 test in 5.712s 2023-01-11T22:14:04.8108131Z 2023-01-11T22:14:04.8108207Z OK 2023-01-11T22:14:04.8108339Z 2023-01-11T22:14:04.8108464Z Generating XML reports... 2023-01-11T22:14:04.8109069Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214555.xml 2023-01-11T22:14:04.8109779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8110208Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8110774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8111243Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8111467Z 2023-01-11T22:14:04.8111559Z Running tests... 2023-01-11T22:14:04.8111959Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8112483Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8113031Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8114066Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.624s) 2023-01-11T22:14:04.8114583Z 2023-01-11T22:14:04.8114912Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8115234Z Ran 1 test in 1.624s 2023-01-11T22:14:04.8115393Z 2023-01-11T22:14:04.8115499Z OK (skipped=1) 2023-01-11T22:14:04.8115635Z 2023-01-11T22:14:04.8115757Z Generating XML reports... 2023-01-11T22:14:04.8116355Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214604.xml 2023-01-11T22:14:04.8117060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8117485Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8118051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8118511Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8118736Z 2023-01-11T22:14:04.8118843Z Running tests... 2023-01-11T22:14:04.8119225Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8119744Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8120284Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8120796Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29087 2023-01-11T22:14:04.8121273Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29088 2023-01-11T22:14:04.8121878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8122317Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8122866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8123324Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8123895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8124327Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8124869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8125320Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8125765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8126236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8126881Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8127556Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8128075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8128520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8128853Z ok (4.229s) 2023-01-11T22:14:04.8128998Z 2023-01-11T22:14:04.8129263Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8129567Z Ran 1 test in 4.229s 2023-01-11T22:14:04.8129729Z 2023-01-11T22:14:04.8129824Z OK 2023-01-11T22:14:04.8129955Z 2023-01-11T22:14:04.8130078Z Generating XML reports... 2023-01-11T22:14:04.8130928Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214608.xml 2023-01-11T22:14:04.8131619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8132064Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8132719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8133180Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8133388Z 2023-01-11T22:14:04.8133494Z Running tests... 2023-01-11T22:14:04.8133891Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8134416Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8134950Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8135998Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.609s) 2023-01-11T22:14:04.8136514Z 2023-01-11T22:14:04.8136775Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8137096Z Ran 1 test in 1.609s 2023-01-11T22:14:04.8137254Z 2023-01-11T22:14:04.8137343Z OK (skipped=1) 2023-01-11T22:14:04.8137493Z 2023-01-11T22:14:04.8137614Z Generating XML reports... 2023-01-11T22:14:04.8138279Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214615.xml 2023-01-11T22:14:04.8138995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8139420Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8139982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8140439Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8140666Z 2023-01-11T22:14:04.8140774Z Running tests... 2023-01-11T22:14:04.8141156Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8141670Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8142186Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8142672Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29230 2023-01-11T22:14:04.8143114Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29231 2023-01-11T22:14:04.8143715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8144200Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8144755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8145220Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8145789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8146209Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8146774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8147230Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8147675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8148146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8148794Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8149559Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8150068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8150509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8150843Z ok (5.749s) 2023-01-11T22:14:04.8150989Z 2023-01-11T22:14:04.8151258Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8151562Z Ran 1 test in 5.749s 2023-01-11T22:14:04.8151719Z 2023-01-11T22:14:04.8151810Z OK 2023-01-11T22:14:04.8151943Z 2023-01-11T22:14:04.8152064Z Generating XML reports... 2023-01-11T22:14:04.8152652Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214619.xml 2023-01-11T22:14:04.8153358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8153803Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8154367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8154809Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8155033Z 2023-01-11T22:14:04.8155139Z Running tests... 2023-01-11T22:14:04.8155590Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8156105Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8156601Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s) 2023-01-11T22:14:04.8156884Z 2023-01-11T22:14:04.8157144Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8157464Z Ran 1 test in 0.002s 2023-01-11T22:14:04.8157626Z 2023-01-11T22:14:04.8157717Z OK (skipped=1) 2023-01-11T22:14:04.8157870Z 2023-01-11T22:14:04.8157992Z Generating XML reports... 2023-01-11T22:14:04.8158594Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214627.xml 2023-01-11T22:14:04.8159297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8159724Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8160292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8160753Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8160983Z 2023-01-11T22:14:04.8161074Z Running tests... 2023-01-11T22:14:04.8161472Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8161993Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8162452Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.8162905Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8163378Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29374 2023-01-11T22:14:04.8163828Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29375 2023-01-11T22:14:04.8164415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8164864Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8165431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8165892Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8166508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8166948Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8167517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8167986Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8168417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8168909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8169561Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8170225Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8170983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8171453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8171797Z ok (4.271s) 2023-01-11T22:14:04.8171944Z 2023-01-11T22:14:04.8172197Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8172525Z Ran 1 test in 4.272s 2023-01-11T22:14:04.8172684Z 2023-01-11T22:14:04.8172853Z OK 2023-01-11T22:14:04.8172996Z 2023-01-11T22:14:04.8173102Z Generating XML reports... 2023-01-11T22:14:04.8173711Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214629.xml 2023-01-11T22:14:04.8174417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8174860Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8175410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8175874Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8176099Z 2023-01-11T22:14:04.8176205Z Running tests... 2023-01-11T22:14:04.8176607Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8177113Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8177590Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.8178085Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8178546Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29487 2023-01-11T22:14:04.8178984Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29488 2023-01-11T22:14:04.8179585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8180031Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8180566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8181005Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8181571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8182034Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8182592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8183049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8183494Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8184049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8184699Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8185374Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8185889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8186337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8186810Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.8187289Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.8187620Z ok (4.270s) 2023-01-11T22:14:04.8187771Z 2023-01-11T22:14:04.8188038Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8188366Z Ran 1 test in 4.270s 2023-01-11T22:14:04.8188525Z 2023-01-11T22:14:04.8188618Z OK 2023-01-11T22:14:04.8188731Z 2023-01-11T22:14:04.8188859Z Generating XML reports... 2023-01-11T22:14:04.8189463Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214636.xml 2023-01-11T22:14:04.8190224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8190676Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8191227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8191690Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8191918Z 2023-01-11T22:14:04.8192027Z Running tests... 2023-01-11T22:14:04.8192417Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8192945Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8193436Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.8193966Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8194449Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29600 2023-01-11T22:14:04.8194891Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29601 2023-01-11T22:14:04.8195489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8195914Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8196485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8196956Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8197526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8197940Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8198512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8198970Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8199422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8199889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8200541Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8201294Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8201786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8202248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8202715Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.8203193Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.8203526Z ok (4.244s) 2023-01-11T22:14:04.8203673Z 2023-01-11T22:14:04.8203945Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8204266Z Ran 1 test in 4.244s 2023-01-11T22:14:04.8204426Z 2023-01-11T22:14:04.8204501Z OK 2023-01-11T22:14:04.8204634Z 2023-01-11T22:14:04.8204763Z Generating XML reports... 2023-01-11T22:14:04.8205366Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214643.xml 2023-01-11T22:14:04.8206072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8206499Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8207125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8207599Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8207823Z 2023-01-11T22:14:04.8207932Z Running tests... 2023-01-11T22:14:04.8208324Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8208847Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8209322Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn) 2023-01-11T22:14:04.8209798Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8210272Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29713 2023-01-11T22:14:04.8210954Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29714 2023-01-11T22:14:04.8211572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8212000Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8212567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8213026Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8213580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8214029Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8214600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8215063Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8215493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8215993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8216645Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8217328Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8217815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8218373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8218717Z ok (4.254s) 2023-01-11T22:14:04.8218864Z 2023-01-11T22:14:04.8219115Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8219443Z Ran 1 test in 4.254s 2023-01-11T22:14:04.8219602Z 2023-01-11T22:14:04.8219695Z OK 2023-01-11T22:14:04.8219828Z 2023-01-11T22:14:04.8219951Z Generating XML reports... 2023-01-11T22:14:04.8220538Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214650.xml 2023-01-11T22:14:04.8221255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8221702Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8222273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8222724Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8222951Z 2023-01-11T22:14:04.8223063Z Running tests... 2023-01-11T22:14:04.8223466Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8223968Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8224538Z test_all_gather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8225031Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29826 2023-01-11T22:14:04.8225476Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29827 2023-01-11T22:14:04.8226062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8226508Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8227083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8227527Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8228104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8228539Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8229109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8229554Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8230002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8230491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8231143Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8231809Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8232329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8232792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8233353Z STAGE:2023-01-11 21:47:01 29827:29827 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8233920Z STAGE:2023-01-11 21:47:01 29826:29826 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8234496Z STAGE:2023-01-11 21:47:01 29827:29827 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8235077Z STAGE:2023-01-11 21:47:01 29826:29826 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8235720Z STAGE:2023-01-11 21:47:01 29827:29827 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8236315Z STAGE:2023-01-11 21:47:01 29826:29826 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8236884Z STAGE:2023-01-11 21:47:01 29827:29827 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8237444Z STAGE:2023-01-11 21:47:01 29826:29826 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8237999Z STAGE:2023-01-11 21:47:01 29827:29827 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8238566Z STAGE:2023-01-11 21:47:01 29826:29826 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8239151Z STAGE:2023-01-11 21:47:01 29827:29827 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8239742Z STAGE:2023-01-11 21:47:01 29826:29826 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8240084Z ok (4.279s) 2023-01-11T22:14:04.8240230Z 2023-01-11T22:14:04.8240491Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8240808Z Ran 1 test in 4.279s 2023-01-11T22:14:04.8240967Z 2023-01-11T22:14:04.8241043Z OK 2023-01-11T22:14:04.8241179Z 2023-01-11T22:14:04.8241303Z Generating XML reports... 2023-01-11T22:14:04.8241954Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214657.xml 2023-01-11T22:14:04.8242680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8243103Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8243676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8244180Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8244415Z 2023-01-11T22:14:04.8244505Z Running tests... 2023-01-11T22:14:04.8244910Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8245438Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8245963Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8246443Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29939 2023-01-11T22:14:04.8246884Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29940 2023-01-11T22:14:04.8247479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8247925Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8248475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8248941Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8249518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8249934Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8250730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8251198Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8251645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8252116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8252777Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8253609Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8254128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8254577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8255156Z STAGE:2023-01-11 21:47:07 29940:29940 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8255728Z STAGE:2023-01-11 21:47:07 29939:29939 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8256696Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2588: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T22:14:04.8257289Z warnings.warn( 2023-01-11T22:14:04.8258149Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2588: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T22:14:04.8258757Z warnings.warn( 2023-01-11T22:14:04.8259231Z STAGE:2023-01-11 21:47:07 29940:29940 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8259860Z STAGE:2023-01-11 21:47:07 29939:29939 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8260468Z STAGE:2023-01-11 21:47:07 29940:29940 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8261066Z STAGE:2023-01-11 21:47:07 29939:29939 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8261640Z STAGE:2023-01-11 21:47:07 29940:29940 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8262182Z STAGE:2023-01-11 21:47:07 29939:29939 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8262755Z STAGE:2023-01-11 21:47:07 29940:29940 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8263326Z STAGE:2023-01-11 21:47:07 29939:29939 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8263896Z STAGE:2023-01-11 21:47:07 29940:29940 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8264492Z STAGE:2023-01-11 21:47:07 29939:29939 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8265059Z STAGE:2023-01-11 21:47:07 29940:29940 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8265619Z STAGE:2023-01-11 21:47:07 29939:29939 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8266163Z STAGE:2023-01-11 21:47:07 29940:29940 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8266968Z STAGE:2023-01-11 21:47:07 29939:29939 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:47:07 29940:29940 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8267357Z 2023-01-11T22:14:04.8267703Z STAGE:2023-01-11 21:47:07 29939:29939 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8268058Z ok (4.237s) 2023-01-11T22:14:04.8268205Z 2023-01-11T22:14:04.8268454Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8268781Z Ran 1 test in 4.237s 2023-01-11T22:14:04.8268940Z 2023-01-11T22:14:04.8269033Z OK 2023-01-11T22:14:04.8269171Z 2023-01-11T22:14:04.8269295Z Generating XML reports... 2023-01-11T22:14:04.8269879Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214704.xml 2023-01-11T22:14:04.8270587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8271107Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8271658Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8272132Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8272357Z 2023-01-11T22:14:04.8272462Z Running tests... 2023-01-11T22:14:04.8272863Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8273369Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8273899Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8274394Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30052 2023-01-11T22:14:04.8274839Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30053 2023-01-11T22:14:04.8275427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8275873Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8276443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8276946Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8277531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8277965Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8278529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8278976Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8279422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8279910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8280553Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8281213Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8281736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8282196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8282657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T22:14:04.8283142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T22:14:04.8283787Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.8284460Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.8285052Z STAGE:2023-01-11 21:47:14 30053:30053 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8285623Z STAGE:2023-01-11 21:47:14 30052:30052 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8286594Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2588: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T22:14:04.8287216Z warnings.warn( 2023-01-11T22:14:04.8288048Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2588: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T22:14:04.8288728Z warnings.warn( 2023-01-11T22:14:04.8289209Z STAGE:2023-01-11 21:47:14 30053:30053 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8289792Z STAGE:2023-01-11 21:47:14 30052:30052 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8290606Z STAGE:2023-01-11 21:47:14 30053:30053 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8291219Z STAGE:2023-01-11 21:47:14 30052:30052 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8291792Z STAGE:2023-01-11 21:47:14 30053:30053 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8292354Z STAGE:2023-01-11 21:47:14 30052:30052 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8292912Z STAGE:2023-01-11 21:47:14 30052:30052 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8293493Z STAGE:2023-01-11 21:47:14 30053:30053 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8294075Z STAGE:2023-01-11 21:47:14 30052:30052 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8294748Z STAGE:2023-01-11 21:47:14 30053:30053 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8295319Z STAGE:2023-01-11 21:47:14 30053:30053 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8295879Z STAGE:2023-01-11 21:47:14 30052:30052 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8296443Z STAGE:2023-01-11 21:47:14 30053:30053 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8297023Z STAGE:2023-01-11 21:47:14 30052:30052 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8297586Z STAGE:2023-01-11 21:47:14 30053:30053 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8298179Z STAGE:2023-01-11 21:47:14 30052:30052 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8298522Z ok (4.251s) 2023-01-11T22:14:04.8298666Z 2023-01-11T22:14:04.8298920Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8299244Z Ran 1 test in 4.251s 2023-01-11T22:14:04.8299407Z 2023-01-11T22:14:04.8299499Z OK 2023-01-11T22:14:04.8299632Z 2023-01-11T22:14:04.8299752Z Generating XML reports... 2023-01-11T22:14:04.8300342Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214710.xml 2023-01-11T22:14:04.8301058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8301502Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8302050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8302515Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8302742Z 2023-01-11T22:14:04.8302850Z Running tests... 2023-01-11T22:14:04.8303246Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8303757Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8304273Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8304759Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30171 2023-01-11T22:14:04.8305187Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30172 2023-01-11T22:14:04.8305872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8306313Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8306879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8307331Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8307907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8308345Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8308913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8309359Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8309806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8310304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8310936Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8311626Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8312244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8312722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8313098Z skip: Skipped due to small world size. (4.218s) 2023-01-11T22:14:04.8313300Z 2023-01-11T22:14:04.8313569Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8313896Z Ran 1 test in 4.218s 2023-01-11T22:14:04.8314060Z 2023-01-11T22:14:04.8314167Z OK (skipped=1) 2023-01-11T22:14:04.8314302Z 2023-01-11T22:14:04.8314427Z Generating XML reports... 2023-01-11T22:14:04.8315035Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214717.xml 2023-01-11T22:14:04.8315740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8316173Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8316745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8317211Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8317438Z 2023-01-11T22:14:04.8317543Z Running tests... 2023-01-11T22:14:04.8317924Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8318445Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8318960Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8319434Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30280 2023-01-11T22:14:04.8319875Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30281 2023-01-11T22:14:04.8320475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8320919Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8321469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8321927Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8322498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8322991Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8323542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8323994Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8324439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8324919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8325570Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8326249Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8326762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8327210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8327777Z STAGE:2023-01-11 21:47:28 30281:30281 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8328350Z STAGE:2023-01-11 21:47:28 30280:30280 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8329367Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2588: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T22:14:04.8329965Z warnings.warn( 2023-01-11T22:14:04.8331058Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2588: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T22:14:04.8331675Z warnings.warn( 2023-01-11T22:14:04.8332152Z STAGE:2023-01-11 21:47:28 30281:30281 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8332713Z STAGE:2023-01-11 21:47:28 30280:30280 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8333301Z STAGE:2023-01-11 21:47:28 30281:30281 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8333903Z STAGE:2023-01-11 21:47:28 30280:30280 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8334477Z STAGE:2023-01-11 21:47:28 30281:30281 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8335019Z STAGE:2023-01-11 21:47:28 30280:30280 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8335592Z STAGE:2023-01-11 21:47:28 30281:30281 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8336176Z STAGE:2023-01-11 21:47:28 30280:30280 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8336741Z STAGE:2023-01-11 21:47:28 30281:30281 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8337331Z STAGE:2023-01-11 21:47:28 30280:30280 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8337911Z STAGE:2023-01-11 21:47:28 30281:30281 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8338470Z STAGE:2023-01-11 21:47:28 30280:30280 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8339021Z STAGE:2023-01-11 21:47:28 30281:30281 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8339590Z STAGE:2023-01-11 21:47:28 30280:30280 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8340163Z STAGE:2023-01-11 21:47:28 30281:30281 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8340843Z STAGE:2023-01-11 21:47:28 30280:30280 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8341172Z ok (4.260s) 2023-01-11T22:14:04.8341317Z 2023-01-11T22:14:04.8341582Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8341906Z Ran 1 test in 4.261s 2023-01-11T22:14:04.8342066Z 2023-01-11T22:14:04.8342142Z OK 2023-01-11T22:14:04.8342273Z 2023-01-11T22:14:04.8342399Z Generating XML reports... 2023-01-11T22:14:04.8343000Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214724.xml 2023-01-11T22:14:04.8343710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8344186Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8344754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8345221Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8345445Z 2023-01-11T22:14:04.8345537Z Running tests... 2023-01-11T22:14:04.8345933Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8346451Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8347034Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8347525Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30393 2023-01-11T22:14:04.8347964Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30394 2023-01-11T22:14:04.8348563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8348985Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8349555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8350012Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8350582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8351003Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8351562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8352015Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8352455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8352922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8353576Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8354254Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8354744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8355209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8355974Z STAGE:2023-01-11 21:47:35 30394:30394 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:47:35 30393:30393 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8356344Z 2023-01-11T22:14:04.8357074Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2588: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T22:14:04.8357746Z warnings.warn( 2023-01-11T22:14:04.8358584Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2588: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T22:14:04.8359182Z warnings.warn( 2023-01-11T22:14:04.8359655Z STAGE:2023-01-11 21:47:35 30394:30394 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8360228Z STAGE:2023-01-11 21:47:35 30393:30393 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8360793Z STAGE:2023-01-11 21:47:35 30394:30394 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8361386Z STAGE:2023-01-11 21:47:35 30393:30393 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8361732Z ok (4.262s) 2023-01-11T22:14:04.8361877Z 2023-01-11T22:14:04.8362125Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8362444Z Ran 1 test in 4.262s 2023-01-11T22:14:04.8362602Z 2023-01-11T22:14:04.8362696Z OK 2023-01-11T22:14:04.8362828Z 2023-01-11T22:14:04.8362948Z Generating XML reports... 2023-01-11T22:14:04.8363582Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214731.xml 2023-01-11T22:14:04.8364298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8399764Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8400491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8400976Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8401229Z 2023-01-11T22:14:04.8401339Z Running tests... 2023-01-11T22:14:04.8401748Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8402303Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8402819Z test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8403312Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30506 2023-01-11T22:14:04.8403756Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30507 2023-01-11T22:14:04.8404368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8404818Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8405481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8405959Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8406544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8406991Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8407556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8408028Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8408477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8408970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8409629Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8410735Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8411263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8411723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8412317Z STAGE:2023-01-11 21:47:42 30506:30506 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8412897Z STAGE:2023-01-11 21:47:42 30507:30507 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8413478Z STAGE:2023-01-11 21:47:42 30507:30507 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8414061Z STAGE:2023-01-11 21:47:42 30506:30506 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8414898Z STAGE:2023-01-11 21:47:42 30507:30507 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:47:42 30506:30506 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8415320Z 2023-01-11T22:14:04.8415654Z STAGE:2023-01-11 21:47:42 30507:30507 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8416231Z STAGE:2023-01-11 21:47:42 30506:30506 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8416894Z STAGE:2023-01-11 21:47:42 30507:30507 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8417504Z STAGE:2023-01-11 21:47:42 30506:30506 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8418100Z STAGE:2023-01-11 21:47:42 30507:30507 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8418714Z STAGE:2023-01-11 21:47:42 30506:30506 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8419055Z ok (4.251s) 2023-01-11T22:14:04.8419202Z 2023-01-11T22:14:04.8419473Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8419788Z Ran 1 test in 4.251s 2023-01-11T22:14:04.8419942Z 2023-01-11T22:14:04.8420017Z OK 2023-01-11T22:14:04.8420144Z 2023-01-11T22:14:04.8420260Z Generating XML reports... 2023-01-11T22:14:04.8420868Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214738.xml 2023-01-11T22:14:04.8421598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8422040Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8422614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8423081Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8423305Z 2023-01-11T22:14:04.8423406Z Running tests... 2023-01-11T22:14:04.8423803Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8424332Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8424841Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s) 2023-01-11T22:14:04.8425134Z 2023-01-11T22:14:04.8425394Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8425713Z Ran 1 test in 0.002s 2023-01-11T22:14:04.8425867Z 2023-01-11T22:14:04.8425964Z OK (skipped=1) 2023-01-11T22:14:04.8426110Z 2023-01-11T22:14:04.8426223Z Generating XML reports... 2023-01-11T22:14:04.8426828Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214744.xml 2023-01-11T22:14:04.8427549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8428081Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8428657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8429124Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8429351Z 2023-01-11T22:14:04.8429449Z Running tests... 2023-01-11T22:14:04.8429846Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8430371Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8430893Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s) 2023-01-11T22:14:04.8431196Z 2023-01-11T22:14:04.8431466Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8431776Z Ran 1 test in 0.002s 2023-01-11T22:14:04.8431931Z 2023-01-11T22:14:04.8432029Z OK (skipped=1) 2023-01-11T22:14:04.8432180Z 2023-01-11T22:14:04.8432293Z Generating XML reports... 2023-01-11T22:14:04.8432914Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214747.xml 2023-01-11T22:14:04.8433629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8434084Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8434723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8435210Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8435430Z 2023-01-11T22:14:04.8435537Z Running tests... 2023-01-11T22:14:04.8435946Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8436497Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8437002Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8437502Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30685 2023-01-11T22:14:04.8437970Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30686 2023-01-11T22:14:04.8438580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8439026Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8439601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8440066Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8440634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8441078Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8441662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8442138Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8442573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8443244Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8443792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8444503Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8445034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8445512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8446057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T22:14:04.8446555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T22:14:04.8447225Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.8447932Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.8448577Z STAGE:2023-01-11 21:47:53 30685:30685 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8449152Z STAGE:2023-01-11 21:47:53 30686:30686 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8449744Z STAGE:2023-01-11 21:47:53 30685:30685 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8450675Z STAGE:2023-01-11 21:47:53 30686:30686 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8451552Z STAGE:2023-01-11 21:47:53 30685:30685 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:47:53 30686:30686 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8451967Z 2023-01-11T22:14:04.8452387Z STAGE:2023-01-11 21:47:53 30686:30686 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8452990Z STAGE:2023-01-11 21:47:53 30685:30685 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8453595Z STAGE:2023-01-11 21:47:53 30685:30685 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8454197Z STAGE:2023-01-11 21:47:53 30686:30686 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8454820Z STAGE:2023-01-11 21:47:53 30685:30685 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8455463Z STAGE:2023-01-11 21:47:53 30686:30686 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8455830Z ok (4.270s) 2023-01-11T22:14:04.8455983Z 2023-01-11T22:14:04.8456262Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8456578Z Ran 1 test in 4.270s 2023-01-11T22:14:04.8456743Z 2023-01-11T22:14:04.8456826Z OK 2023-01-11T22:14:04.8456961Z 2023-01-11T22:14:04.8457082Z Generating XML reports... 2023-01-11T22:14:04.8457693Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214749.xml 2023-01-11T22:14:04.8458420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8458893Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8459484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8459953Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8460191Z 2023-01-11T22:14:04.8460291Z Running tests... 2023-01-11T22:14:04.8460698Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8461233Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8461755Z test_all_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8462250Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30804 2023-01-11T22:14:04.8462713Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30805 2023-01-11T22:14:04.8463335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8463796Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8464470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8464957Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8465543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8466008Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8466601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8467077Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8467539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8468047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8468726Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8469425Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8469956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8470507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8470920Z skip: Skipped due to small world size. (4.169s) 2023-01-11T22:14:04.8471132Z 2023-01-11T22:14:04.8471407Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8471745Z Ran 1 test in 4.169s 2023-01-11T22:14:04.8471905Z 2023-01-11T22:14:04.8472014Z OK (skipped=1) 2023-01-11T22:14:04.8472164Z 2023-01-11T22:14:04.8472273Z Generating XML reports... 2023-01-11T22:14:04.8472901Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214756.xml 2023-01-11T22:14:04.8473647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8474102Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8474680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8475166Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8475408Z 2023-01-11T22:14:04.8475508Z Running tests... 2023-01-11T22:14:04.8475911Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8476451Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8477011Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s) 2023-01-11T22:14:04.8477350Z 2023-01-11T22:14:04.8477625Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8477937Z Ran 1 test in 0.002s 2023-01-11T22:14:04.8478103Z 2023-01-11T22:14:04.8478212Z OK (skipped=1) 2023-01-11T22:14:04.8478362Z 2023-01-11T22:14:04.8478486Z Generating XML reports... 2023-01-11T22:14:04.8479086Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214803.xml 2023-01-11T22:14:04.8479822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8480281Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8480878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8481341Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8481641Z 2023-01-11T22:14:04.8481741Z Running tests... 2023-01-11T22:14:04.8482165Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8482689Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8483253Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s) 2023-01-11T22:14:04.8483598Z 2023-01-11T22:14:04.8483872Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8484198Z Ran 1 test in 0.002s 2023-01-11T22:14:04.8484364Z 2023-01-11T22:14:04.8484456Z OK (skipped=1) 2023-01-11T22:14:04.8484615Z 2023-01-11T22:14:04.8484729Z Generating XML reports... 2023-01-11T22:14:04.8485355Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214805.xml 2023-01-11T22:14:04.8486092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8486540Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8487142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8487625Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8487867Z 2023-01-11T22:14:04.8487960Z Running tests... 2023-01-11T22:14:04.8488424Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8488976Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8489525Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s) 2023-01-11T22:14:04.8489545Z 2023-01-11T22:14:04.8489802Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8489921Z Ran 1 test in 0.002s 2023-01-11T22:14:04.8489941Z 2023-01-11T22:14:04.8490049Z OK (skipped=1) 2023-01-11T22:14:04.8490068Z 2023-01-11T22:14:04.8490182Z Generating XML reports... 2023-01-11T22:14:04.8490843Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214808.xml 2023-01-11T22:14:04.8491216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8491384Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8491757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8491934Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8491955Z 2023-01-11T22:14:04.8492046Z Running tests... 2023-01-11T22:14:04.8492309Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8492624Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8492906Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s) 2023-01-11T22:14:04.8492925Z 2023-01-11T22:14:04.8493181Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8493280Z Ran 1 test in 0.002s 2023-01-11T22:14:04.8493300Z 2023-01-11T22:14:04.8493403Z OK (skipped=1) 2023-01-11T22:14:04.8493425Z 2023-01-11T22:14:04.8493536Z Generating XML reports... 2023-01-11T22:14:04.8493982Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214810.xml 2023-01-11T22:14:04.8494333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8494505Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8494973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8495150Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8495170Z 2023-01-11T22:14:04.8495275Z Running tests... 2023-01-11T22:14:04.8495524Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8495830Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8496103Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8496302Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31045 2023-01-11T22:14:04.8496517Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31046 2023-01-11T22:14:04.8496882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8497047Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8497422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8497609Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8497966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8498199Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8498586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8498752Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8498994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8499233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8499620Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8500008Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8500233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8500462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8500553Z ok (4.249s) 2023-01-11T22:14:04.8500574Z 2023-01-11T22:14:04.8500839Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8500933Z Ran 1 test in 4.250s 2023-01-11T22:14:04.8500953Z 2023-01-11T22:14:04.8501043Z OK 2023-01-11T22:14:04.8501062Z 2023-01-11T22:14:04.8501184Z Generating XML reports... 2023-01-11T22:14:04.8501631Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214812.xml 2023-01-11T22:14:04.8501991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8502161Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8502534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8502724Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8502744Z 2023-01-11T22:14:04.8502835Z Running tests... 2023-01-11T22:14:04.8503093Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8503404Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8503670Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8503946Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31154 2023-01-11T22:14:04.8504160Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31155 2023-01-11T22:14:04.8504528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8504696Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8505073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8505243Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8505598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8505768Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8506135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8506311Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8506551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8506789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8507238Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8507625Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8507852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8508073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8508307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T22:14:04.8508545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T22:14:04.8508928Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.8509312Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.8509547Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2023-01-11T22:14:04.8509770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2023-01-11T22:14:04.8510141Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2023-01-11T22:14:04.8510525Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2023-01-11T22:14:04.8510753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2023-01-11T22:14:04.8510982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2023-01-11T22:14:04.8511353Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2023-01-11T22:14:04.8511738Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2023-01-11T22:14:04.8511829Z ok (4.337s) 2023-01-11T22:14:04.8511849Z 2023-01-11T22:14:04.8512108Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8512218Z Ran 1 test in 4.337s 2023-01-11T22:14:04.8512238Z 2023-01-11T22:14:04.8512313Z OK 2023-01-11T22:14:04.8512338Z 2023-01-11T22:14:04.8512444Z Generating XML reports... 2023-01-11T22:14:04.8512965Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214819.xml 2023-01-11T22:14:04.8513329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8513490Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8513861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8514053Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8514072Z 2023-01-11T22:14:04.8514170Z Running tests... 2023-01-11T22:14:04.8514427Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8514717Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8514964Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports all_gather_v (0.002s) 2023-01-11T22:14:04.8514987Z 2023-01-11T22:14:04.8515240Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8515337Z Ran 1 test in 0.003s 2023-01-11T22:14:04.8515356Z 2023-01-11T22:14:04.8515459Z OK (skipped=1) 2023-01-11T22:14:04.8515478Z 2023-01-11T22:14:04.8515587Z Generating XML reports... 2023-01-11T22:14:04.8516073Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214826.xml 2023-01-11T22:14:04.8516436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8516605Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8516961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8517136Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8517159Z 2023-01-11T22:14:04.8517264Z Running tests... 2023-01-11T22:14:04.8517515Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8517825Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8518101Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8518309Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31320 2023-01-11T22:14:04.8518521Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31321 2023-01-11T22:14:04.8518870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8519032Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8519406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8519587Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8519943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8520102Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8520469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8520645Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8520885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8521106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8521497Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8521945Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8522158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8522380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8522614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T22:14:04.8522842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T22:14:04.8523234Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.8523619Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.8523933Z STAGE:2023-01-11 21:48:32 31320:31320 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8524246Z STAGE:2023-01-11 21:48:32 31321:31321 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8524988Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T22:14:04.8525146Z warnings.warn( 2023-01-11T22:14:04.8525872Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T22:14:04.8525978Z warnings.warn( 2023-01-11T22:14:04.8526315Z STAGE:2023-01-11 21:48:32 31320:31320 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8526645Z STAGE:2023-01-11 21:48:32 31321:31321 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8526975Z STAGE:2023-01-11 21:48:32 31320:31320 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8527318Z STAGE:2023-01-11 21:48:32 31321:31321 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8527829Z STAGE:2023-01-11 21:48:32 31320:31320 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:48:32 31321:31321 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8527866Z 2023-01-11T22:14:04.8528179Z STAGE:2023-01-11 21:48:32 31320:31320 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8528491Z STAGE:2023-01-11 21:48:32 31321:31321 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8528828Z STAGE:2023-01-11 21:48:32 31320:31320 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8529160Z STAGE:2023-01-11 21:48:32 31321:31321 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8529258Z ok (4.261s) 2023-01-11T22:14:04.8529277Z 2023-01-11T22:14:04.8529538Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8529649Z Ran 1 test in 4.261s 2023-01-11T22:14:04.8529669Z 2023-01-11T22:14:04.8529751Z OK 2023-01-11T22:14:04.8529769Z 2023-01-11T22:14:04.8529879Z Generating XML reports... 2023-01-11T22:14:04.8530556Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214828.xml 2023-01-11T22:14:04.8530943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8531105Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8531479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8531757Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8531777Z 2023-01-11T22:14:04.8531874Z Running tests... 2023-01-11T22:14:04.8532134Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8532441Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8532704Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8532909Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31439 2023-01-11T22:14:04.8533120Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31440 2023-01-11T22:14:04.8533488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8533648Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8534025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8534213Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8534563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8534774Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8535162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8535347Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8535575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8535811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8536212Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8536591Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8536814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8537042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8537260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T22:14:04.8537485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T22:14:04.8537873Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.8538261Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.8538583Z STAGE:2023-01-11 21:48:39 31440:31440 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8538900Z STAGE:2023-01-11 21:48:39 31439:31439 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8539641Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T22:14:04.8539744Z warnings.warn( 2023-01-11T22:14:04.8540467Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T22:14:04.8540629Z warnings.warn( 2023-01-11T22:14:04.8541154Z STAGE:2023-01-11 21:48:39 31439:31439 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:48:39 31440:31440 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8541182Z 2023-01-11T22:14:04.8541734Z STAGE:2023-01-11 21:48:39 31440:31440 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:48:39 31439:31439 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8541769Z 2023-01-11T22:14:04.8542270Z STAGE:2023-01-11 21:48:39 31440:31440 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:48:39 31439:31439 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8542296Z 2023-01-11T22:14:04.8542607Z STAGE:2023-01-11 21:48:39 31439:31439 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8542937Z STAGE:2023-01-11 21:48:39 31440:31440 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8543279Z STAGE:2023-01-11 21:48:39 31439:31439 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8543610Z STAGE:2023-01-11 21:48:39 31440:31440 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8543709Z ok (4.329s) 2023-01-11T22:14:04.8543727Z 2023-01-11T22:14:04.8544035Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8544183Z Ran 1 test in 4.329s 2023-01-11T22:14:04.8544203Z 2023-01-11T22:14:04.8544295Z OK 2023-01-11T22:14:04.8544314Z 2023-01-11T22:14:04.8544420Z Generating XML reports... 2023-01-11T22:14:04.8544869Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214835.xml 2023-01-11T22:14:04.8545234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8545412Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8545789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8545977Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8545996Z 2023-01-11T22:14:04.8546102Z Running tests... 2023-01-11T22:14:04.8546368Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8546668Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8546937Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8547152Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31558 2023-01-11T22:14:04.8547365Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31559 2023-01-11T22:14:04.8547739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8547910Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8548280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8548466Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8548826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8548979Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8549347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8549531Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8549771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8550133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8550530Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8550921Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8551150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8551374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8551589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T22:14:04.8551825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T22:14:04.8552220Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.8552607Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.8552986Z STAGE:2023-01-11 21:48:46 31559:31559 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8553361Z STAGE:2023-01-11 21:48:46 31558:31558 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8554100Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T22:14:04.8554210Z warnings.warn( 2023-01-11T22:14:04.8554935Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T22:14:04.8555047Z warnings.warn( 2023-01-11T22:14:04.8555362Z STAGE:2023-01-11 21:48:46 31558:31558 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8555683Z STAGE:2023-01-11 21:48:46 31559:31559 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8556246Z STAGE:2023-01-11 21:48:46 31558:31558 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:48:46 31559:31559 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8556266Z 2023-01-11T22:14:04.8556589Z STAGE:2023-01-11 21:48:46 31558:31558 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8556904Z STAGE:2023-01-11 21:48:46 31559:31559 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8557238Z STAGE:2023-01-11 21:48:46 31558:31558 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8557566Z STAGE:2023-01-11 21:48:46 31559:31559 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8557908Z STAGE:2023-01-11 21:48:46 31558:31558 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8558251Z STAGE:2023-01-11 21:48:46 31559:31559 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8558352Z ok (4.220s) 2023-01-11T22:14:04.8558371Z 2023-01-11T22:14:04.8558615Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8558725Z Ran 1 test in 4.220s 2023-01-11T22:14:04.8558745Z 2023-01-11T22:14:04.8558836Z OK 2023-01-11T22:14:04.8558855Z 2023-01-11T22:14:04.8558978Z Generating XML reports... 2023-01-11T22:14:04.8559425Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214842.xml 2023-01-11T22:14:04.8559853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8560025Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8560401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8560576Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8560613Z 2023-01-11T22:14:04.8560702Z Running tests... 2023-01-11T22:14:04.8560963Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8561275Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8561554Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8561770Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31677 2023-01-11T22:14:04.8561986Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31678 2023-01-11T22:14:04.8562349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8562523Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8562928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8563123Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8563484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8563653Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8564020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8564207Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8564450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8564691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8565070Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8565465Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8565691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8565913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8566146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T22:14:04.8566386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T22:14:04.8566778Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.8567161Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.8567493Z STAGE:2023-01-11 21:48:53 31677:31677 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8567796Z STAGE:2023-01-11 21:48:53 31678:31678 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8568525Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T22:14:04.8568690Z warnings.warn( 2023-01-11T22:14:04.8569415Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T22:14:04.8569523Z warnings.warn( 2023-01-11T22:14:04.8569860Z STAGE:2023-01-11 21:48:53 31677:31677 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8570183Z STAGE:2023-01-11 21:48:53 31678:31678 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8571086Z STAGE:2023-01-11 21:48:53 31677:31677 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:48:53 31678:31678 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8571109Z 2023-01-11T22:14:04.8571433Z STAGE:2023-01-11 21:48:53 31677:31677 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8571757Z STAGE:2023-01-11 21:48:53 31678:31678 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8572086Z STAGE:2023-01-11 21:48:53 31677:31677 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8572397Z STAGE:2023-01-11 21:48:53 31678:31678 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8572819Z STAGE:2023-01-11 21:48:53 31677:31677 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8573176Z STAGE:2023-01-11 21:48:53 31678:31678 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8573277Z ok (4.261s) 2023-01-11T22:14:04.8573297Z 2023-01-11T22:14:04.8573557Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8573668Z Ran 1 test in 4.261s 2023-01-11T22:14:04.8573687Z 2023-01-11T22:14:04.8573784Z OK 2023-01-11T22:14:04.8573803Z 2023-01-11T22:14:04.8573924Z Generating XML reports... 2023-01-11T22:14:04.8574371Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214849.xml 2023-01-11T22:14:04.8574721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8574895Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8575275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8575464Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8575484Z 2023-01-11T22:14:04.8575589Z Running tests... 2023-01-11T22:14:04.8575852Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8576160Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8576435Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8576634Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31796 2023-01-11T22:14:04.8576848Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31797 2023-01-11T22:14:04.8577216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8577391Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8577765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8577953Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8578310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8578573Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8578944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8579114Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8579353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8579596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8579993Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8580381Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8580604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8580833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8580989Z skip: Skipped due to small world size. (4.116s) 2023-01-11T22:14:04.8581009Z 2023-01-11T22:14:04.8581269Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8581365Z Ran 1 test in 4.116s 2023-01-11T22:14:04.8581385Z 2023-01-11T22:14:04.8581490Z OK (skipped=1) 2023-01-11T22:14:04.8581508Z 2023-01-11T22:14:04.8581629Z Generating XML reports... 2023-01-11T22:14:04.8582124Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214856.xml 2023-01-11T22:14:04.8582500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8582674Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8583048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8583241Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8583260Z 2023-01-11T22:14:04.8583351Z Running tests... 2023-01-11T22:14:04.8583613Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8583921Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8584200Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8584417Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31905 2023-01-11T22:14:04.8584629Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31906 2023-01-11T22:14:04.8584996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8585165Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8585547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8585716Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8586074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8586244Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8586612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8586794Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8587035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8587273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8587729Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8588103Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8588330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8588551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8588711Z skip: Skipped due to small world size. (4.157s) 2023-01-11T22:14:04.8588731Z 2023-01-11T22:14:04.8588995Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8589105Z Ran 1 test in 4.157s 2023-01-11T22:14:04.8589124Z 2023-01-11T22:14:04.8589229Z OK (skipped=1) 2023-01-11T22:14:04.8589248Z 2023-01-11T22:14:04.8589369Z Generating XML reports... 2023-01-11T22:14:04.8589813Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214902.xml 2023-01-11T22:14:04.8590164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8590336Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8590710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8590944Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8590964Z 2023-01-11T22:14:04.8591076Z Running tests... 2023-01-11T22:14:04.8591337Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8591647Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8591926Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8592145Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32014 2023-01-11T22:14:04.8592341Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32015 2023-01-11T22:14:04.8592709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8592879Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8593257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8593442Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8593802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8593969Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8594335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8594504Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8594744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8594982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8595377Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8595767Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8595994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8596214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8596369Z skip: Skipped due to small world size. (4.249s) 2023-01-11T22:14:04.8596438Z 2023-01-11T22:14:04.8596707Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8596801Z Ran 1 test in 4.249s 2023-01-11T22:14:04.8596821Z 2023-01-11T22:14:04.8596926Z OK (skipped=1) 2023-01-11T22:14:04.8596945Z 2023-01-11T22:14:04.8597066Z Generating XML reports... 2023-01-11T22:14:04.8597511Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214909.xml 2023-01-11T22:14:04.8597878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8598049Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8598422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8598607Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8598630Z 2023-01-11T22:14:04.8598738Z Running tests... 2023-01-11T22:14:04.8598983Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8599294Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8599564Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8599779Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32123 2023-01-11T22:14:04.8600037Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32124 2023-01-11T22:14:04.8600412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8600582Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8600953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8601127Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8601488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8601654Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8602025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8602215Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8602455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8602693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8603085Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8603479Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8603687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8603911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8604068Z skip: Skipped due to small world size. (4.124s) 2023-01-11T22:14:04.8604087Z 2023-01-11T22:14:04.8604353Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8604464Z Ran 1 test in 4.124s 2023-01-11T22:14:04.8604483Z 2023-01-11T22:14:04.8604589Z OK (skipped=1) 2023-01-11T22:14:04.8604607Z 2023-01-11T22:14:04.8604727Z Generating XML reports... 2023-01-11T22:14:04.8605177Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214916.xml 2023-01-11T22:14:04.8605543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8605761Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8606138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8606322Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8606343Z 2023-01-11T22:14:04.8606450Z Running tests... 2023-01-11T22:14:04.8606712Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8607020Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8607284Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8607499Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32232 2023-01-11T22:14:04.8607697Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32233 2023-01-11T22:14:04.8608068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8608239Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8608608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8608793Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8609199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8609377Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8609744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8609927Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8610154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8610615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8611027Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8611420Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8611646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8611869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8612205Z STAGE:2023-01-11 21:49:26 32233:32233 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8612522Z STAGE:2023-01-11 21:49:26 32232:32232 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8613259Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T22:14:04.8613357Z warnings.warn( 2023-01-11T22:14:04.8614079Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T22:14:04.8614190Z warnings.warn( 2023-01-11T22:14:04.8614729Z STAGE:2023-01-11 21:49:26 32233:32233 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:49:26 32232:32232 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8614750Z 2023-01-11T22:14:04.8615192Z STAGE:2023-01-11 21:49:26 32233:32233 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8615535Z STAGE:2023-01-11 21:49:26 32232:32232 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8615856Z STAGE:2023-01-11 21:49:26 32233:32233 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8616178Z STAGE:2023-01-11 21:49:26 32232:32232 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8616508Z STAGE:2023-01-11 21:49:26 32233:32233 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8616832Z STAGE:2023-01-11 21:49:26 32232:32232 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8617154Z STAGE:2023-01-11 21:49:26 32233:32233 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8617494Z STAGE:2023-01-11 21:49:26 32232:32232 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8617599Z ok (4.266s) 2023-01-11T22:14:04.8617618Z 2023-01-11T22:14:04.8617882Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8617992Z Ran 1 test in 4.266s 2023-01-11T22:14:04.8618011Z 2023-01-11T22:14:04.8618102Z OK 2023-01-11T22:14:04.8618121Z 2023-01-11T22:14:04.8618243Z Generating XML reports... 2023-01-11T22:14:04.8618751Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214922.xml 2023-01-11T22:14:04.8619131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8619287Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8619664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8619851Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8619874Z 2023-01-11T22:14:04.8619979Z Running tests... 2023-01-11T22:14:04.8620240Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8620548Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8620844Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8621065Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32345 2023-01-11T22:14:04.8621261Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32346 2023-01-11T22:14:04.8621630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8621801Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8622176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8622371Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8622732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8622902Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8623271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8623456Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8623681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8623921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8624315Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8624772Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8624999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8625224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8625951Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T22:14:04.8626062Z warnings.warn( 2023-01-11T22:14:04.8626784Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T22:14:04.8626896Z warnings.warn( 2023-01-11T22:14:04.8626978Z ok (4.102s) 2023-01-11T22:14:04.8626997Z 2023-01-11T22:14:04.8627257Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8627365Z Ran 1 test in 4.102s 2023-01-11T22:14:04.8627385Z 2023-01-11T22:14:04.8627476Z OK 2023-01-11T22:14:04.8627495Z 2023-01-11T22:14:04.8627616Z Generating XML reports... 2023-01-11T22:14:04.8628107Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214929.xml 2023-01-11T22:14:04.8628483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8628655Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8629010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8629202Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8629222Z 2023-01-11T22:14:04.8629327Z Running tests... 2023-01-11T22:14:04.8629588Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8629898Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8630164Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8630380Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32454 2023-01-11T22:14:04.8630592Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32455 2023-01-11T22:14:04.8630939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8631114Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8631488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8631672Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8632029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8632198Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8632566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8632748Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8632988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8633366Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8633599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8634048Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8634274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8634496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8634832Z STAGE:2023-01-11 21:49:40 32454:32454 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8635152Z STAGE:2023-01-11 21:49:40 32455:32455 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8635879Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T22:14:04.8635995Z warnings.warn( 2023-01-11T22:14:04.8636714Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T22:14:04.8636807Z warnings.warn( 2023-01-11T22:14:04.8637186Z STAGE:2023-01-11 21:49:40 32454:32454 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8637520Z STAGE:2023-01-11 21:49:40 32455:32455 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8637864Z STAGE:2023-01-11 21:49:40 32454:32454 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8638206Z STAGE:2023-01-11 21:49:40 32455:32455 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8638529Z STAGE:2023-01-11 21:49:40 32454:32454 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8638848Z STAGE:2023-01-11 21:49:40 32455:32455 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8639383Z STAGE:2023-01-11 21:49:40 32455:32455 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:49:40 32454:32454 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8639404Z 2023-01-11T22:14:04.8639969Z STAGE:2023-01-11 21:49:40 32454:32454 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:49:40 32455:32455 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8639989Z 2023-01-11T22:14:04.8640089Z ok (4.263s) 2023-01-11T22:14:04.8640109Z 2023-01-11T22:14:04.8640353Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8640465Z Ran 1 test in 4.263s 2023-01-11T22:14:04.8640484Z 2023-01-11T22:14:04.8640578Z OK 2023-01-11T22:14:04.8640597Z 2023-01-11T22:14:04.8640719Z Generating XML reports... 2023-01-11T22:14:04.8641170Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214936.xml 2023-01-11T22:14:04.8641537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8641709Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8642088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8642274Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8642294Z 2023-01-11T22:14:04.8642383Z Running tests... 2023-01-11T22:14:04.8642646Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8642955Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8643295Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8643511Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32567 2023-01-11T22:14:04.8643725Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32568 2023-01-11T22:14:04.8644095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8644312Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8644677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8644864Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8645227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8645401Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8645770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8645953Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8646194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8646479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8646886Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8647260Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8647485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8647711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8648042Z STAGE:2023-01-11 21:49:47 32568:32568 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8648356Z STAGE:2023-01-11 21:49:47 32567:32567 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8649083Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T22:14:04.8649196Z warnings.warn( 2023-01-11T22:14:04.8649918Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T22:14:04.8650032Z warnings.warn( 2023-01-11T22:14:04.8650540Z STAGE:2023-01-11 21:49:47 32567:32567 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8650863Z STAGE:2023-01-11 21:49:47 32568:32568 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8651203Z STAGE:2023-01-11 21:49:47 32567:32567 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8651547Z STAGE:2023-01-11 21:49:47 32568:32568 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8651874Z STAGE:2023-01-11 21:49:47 32567:32567 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8652190Z STAGE:2023-01-11 21:49:47 32568:32568 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8652519Z STAGE:2023-01-11 21:49:47 32568:32568 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8652842Z STAGE:2023-01-11 21:49:47 32567:32567 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8653277Z STAGE:2023-01-11 21:49:47 32568:32568 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8653614Z STAGE:2023-01-11 21:49:47 32567:32567 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8653698Z ok (4.247s) 2023-01-11T22:14:04.8653719Z 2023-01-11T22:14:04.8653978Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8654092Z Ran 1 test in 4.247s 2023-01-11T22:14:04.8654112Z 2023-01-11T22:14:04.8654203Z OK 2023-01-11T22:14:04.8654223Z 2023-01-11T22:14:04.8654345Z Generating XML reports... 2023-01-11T22:14:04.8654793Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214943.xml 2023-01-11T22:14:04.8655160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8655337Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8655697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8655884Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8655903Z 2023-01-11T22:14:04.8656011Z Running tests... 2023-01-11T22:14:04.8656268Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8656649Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8656926Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8657143Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32680 2023-01-11T22:14:04.8657356Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32681 2023-01-11T22:14:04.8657729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8657885Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8658257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8658444Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8658806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8658978Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8659351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8659536Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8659777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8660003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8660401Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8660789Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8661017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8661242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8661571Z STAGE:2023-01-11 21:49:53 32681:32681 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8661890Z STAGE:2023-01-11 21:49:53 32680:32680 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8662620Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T22:14:04.8662784Z warnings.warn( 2023-01-11T22:14:04.8663512Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2023-01-11T22:14:04.8663607Z warnings.warn( 2023-01-11T22:14:04.8663940Z STAGE:2023-01-11 21:49:53 32681:32681 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8664264Z STAGE:2023-01-11 21:49:53 32680:32680 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8664609Z STAGE:2023-01-11 21:49:53 32681:32681 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8664954Z STAGE:2023-01-11 21:49:53 32680:32680 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8665276Z STAGE:2023-01-11 21:49:53 32681:32681 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8665593Z STAGE:2023-01-11 21:49:53 32680:32680 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8665968Z STAGE:2023-01-11 21:49:53 32681:32681 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8666309Z STAGE:2023-01-11 21:49:53 32680:32680 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8666631Z STAGE:2023-01-11 21:49:53 32681:32681 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8666970Z STAGE:2023-01-11 21:49:53 32680:32680 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8667070Z ok (4.262s) 2023-01-11T22:14:04.8667094Z 2023-01-11T22:14:04.8667355Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8667467Z Ran 1 test in 4.263s 2023-01-11T22:14:04.8667486Z 2023-01-11T22:14:04.8667567Z OK 2023-01-11T22:14:04.8667586Z 2023-01-11T22:14:04.8667707Z Generating XML reports... 2023-01-11T22:14:04.8668145Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214949.xml 2023-01-11T22:14:04.8668507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8668667Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8669041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8669225Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8669245Z 2023-01-11T22:14:04.8669355Z Running tests... 2023-01-11T22:14:04.8669616Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8669928Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8670209Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8670423Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32793 2023-01-11T22:14:04.8670621Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32794 2023-01-11T22:14:04.8670990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8671160Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8671534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8671775Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8672139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8672304Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8672669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8672856Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8673082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8673321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8673713Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8674103Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8674332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8674553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8674652Z ok (4.260s) 2023-01-11T22:14:04.8674672Z 2023-01-11T22:14:04.8674930Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8675076Z Ran 1 test in 4.260s 2023-01-11T22:14:04.8675115Z 2023-01-11T22:14:04.8675193Z OK 2023-01-11T22:14:04.8675212Z 2023-01-11T22:14:04.8675334Z Generating XML reports... 2023-01-11T22:14:04.8675781Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214956.xml 2023-01-11T22:14:04.8676147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8676323Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8676698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8676876Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8676896Z 2023-01-11T22:14:04.8676997Z Running tests... 2023-01-11T22:14:04.8677238Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8677544Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8677806Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8678021Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32902 2023-01-11T22:14:04.8678234Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32903 2023-01-11T22:14:04.8678600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8678770Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8679145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8679314Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8679672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8679840Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8680205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8680391Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8680630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8680921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8681318Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8681707Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8681916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8682141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8682375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T22:14:04.8682612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T22:14:04.8683004Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.8683396Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.8683728Z STAGE:2023-01-11 21:50:07 32903:32903 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8684048Z STAGE:2023-01-11 21:50:07 32902:32902 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8684423Z STAGE:2023-01-11 21:50:07 32902:32902 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8684738Z STAGE:2023-01-11 21:50:07 32903:32903 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8685084Z STAGE:2023-01-11 21:50:07 32902:32902 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8685423Z STAGE:2023-01-11 21:50:07 32903:32903 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8685950Z STAGE:2023-01-11 21:50:07 32902:32902 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:50:07 32903:32903 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8685971Z 2023-01-11T22:14:04.8686297Z STAGE:2023-01-11 21:50:07 32902:32902 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8686627Z STAGE:2023-01-11 21:50:07 32903:32903 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8686968Z STAGE:2023-01-11 21:50:07 32902:32902 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8687306Z STAGE:2023-01-11 21:50:07 32903:32903 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8687406Z ok (4.239s) 2023-01-11T22:14:04.8687425Z 2023-01-11T22:14:04.8687684Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8687780Z Ran 1 test in 4.239s 2023-01-11T22:14:04.8687803Z 2023-01-11T22:14:04.8687895Z OK 2023-01-11T22:14:04.8687914Z 2023-01-11T22:14:04.8688040Z Generating XML reports... 2023-01-11T22:14:04.8688486Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215003.xml 2023-01-11T22:14:04.8688848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8689020Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8689395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8689584Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8689603Z 2023-01-11T22:14:04.8689694Z Running tests... 2023-01-11T22:14:04.8689956Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8690266Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8690861Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8691071Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33021 2023-01-11T22:14:04.8691285Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33022 2023-01-11T22:14:04.8691664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8691836Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8692209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8692379Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8692736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8692909Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8693273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8693457Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8693698Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8694049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8694460Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8694833Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8695059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8695287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8695520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T22:14:04.8695755Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T22:14:04.8696149Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.8696536Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.8696865Z STAGE:2023-01-11 21:50:14 33022:33022 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8697187Z STAGE:2023-01-11 21:50:14 33021:33021 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8697499Z STAGE:2023-01-11 21:50:14 33021:33021 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8697827Z STAGE:2023-01-11 21:50:14 33022:33022 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8698166Z STAGE:2023-01-11 21:50:14 33021:33021 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8698510Z STAGE:2023-01-11 21:50:14 33022:33022 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8698837Z STAGE:2023-01-11 21:50:14 33022:33022 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8699153Z STAGE:2023-01-11 21:50:14 33021:33021 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8699480Z STAGE:2023-01-11 21:50:14 33021:33021 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8699799Z STAGE:2023-01-11 21:50:14 33022:33022 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8700199Z STAGE:2023-01-11 21:50:14 33021:33021 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8700517Z STAGE:2023-01-11 21:50:14 33022:33022 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8700618Z ok (4.269s) 2023-01-11T22:14:04.8700638Z 2023-01-11T22:14:04.8700900Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8701010Z Ran 1 test in 4.269s 2023-01-11T22:14:04.8701029Z 2023-01-11T22:14:04.8701125Z OK 2023-01-11T22:14:04.8701144Z 2023-01-11T22:14:04.8701266Z Generating XML reports... 2023-01-11T22:14:04.8701714Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215010.xml 2023-01-11T22:14:04.8702082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8702254Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8702615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8702804Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8702823Z 2023-01-11T22:14:04.8702930Z Running tests... 2023-01-11T22:14:04.8703187Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8703542Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8703821Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8704037Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33140 2023-01-11T22:14:04.8704249Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33141 2023-01-11T22:14:04.8704605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8704786Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8705159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8705344Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8705705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8705879Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8706246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8706429Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8706670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8706895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8707289Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8707678Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8707903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8708128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8708363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T22:14:04.8708602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T22:14:04.8708992Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.8709454Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.8709768Z STAGE:2023-01-11 21:50:21 33140:33140 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8710089Z STAGE:2023-01-11 21:50:21 33141:33141 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8710421Z STAGE:2023-01-11 21:50:21 33140:33140 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8710743Z STAGE:2023-01-11 21:50:21 33141:33141 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8711082Z STAGE:2023-01-11 21:50:21 33140:33140 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8711422Z STAGE:2023-01-11 21:50:21 33141:33141 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8711745Z STAGE:2023-01-11 21:50:21 33140:33140 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8712071Z STAGE:2023-01-11 21:50:21 33141:33141 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8712399Z STAGE:2023-01-11 21:50:21 33141:33141 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8712704Z STAGE:2023-01-11 21:50:21 33140:33140 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8713087Z STAGE:2023-01-11 21:50:21 33141:33141 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8713436Z STAGE:2023-01-11 21:50:21 33140:33140 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8713537Z ok (4.274s) 2023-01-11T22:14:04.8713556Z 2023-01-11T22:14:04.8713815Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8713926Z Ran 1 test in 4.274s 2023-01-11T22:14:04.8713946Z 2023-01-11T22:14:04.8714040Z OK 2023-01-11T22:14:04.8714059Z 2023-01-11T22:14:04.8714181Z Generating XML reports... 2023-01-11T22:14:04.8714626Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215017.xml 2023-01-11T22:14:04.8714975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8715148Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8715524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8715711Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8715731Z 2023-01-11T22:14:04.8715836Z Running tests... 2023-01-11T22:14:04.8716093Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8716402Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8716666Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8716865Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33259 2023-01-11T22:14:04.8717076Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33260 2023-01-11T22:14:04.8717441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8717615Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8717991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8718176Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8718533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8718757Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8719128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8719297Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8719538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8719776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8720171Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8720560Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8720784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8721011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8721243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T22:14:04.8721479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T22:14:04.8721857Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.8722288Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.8722632Z STAGE:2023-01-11 21:50:28 33260:33260 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8722948Z STAGE:2023-01-11 21:50:28 33259:33259 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8723484Z STAGE:2023-01-11 21:50:28 33260:33260 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:50:28 33259:33259 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8723507Z 2023-01-11T22:14:04.8724067Z STAGE:2023-01-11 21:50:28 33260:33260 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:50:28 33259:33259 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8724087Z 2023-01-11T22:14:04.8724412Z STAGE:2023-01-11 21:50:28 33260:33260 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8724728Z STAGE:2023-01-11 21:50:28 33259:33259 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8725054Z STAGE:2023-01-11 21:50:28 33260:33260 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8725383Z STAGE:2023-01-11 21:50:28 33259:33259 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8725705Z STAGE:2023-01-11 21:50:28 33260:33260 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8726052Z STAGE:2023-01-11 21:50:28 33259:33259 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8726152Z ok (4.266s) 2023-01-11T22:14:04.8726172Z 2023-01-11T22:14:04.8726432Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8726542Z Ran 1 test in 4.266s 2023-01-11T22:14:04.8726561Z 2023-01-11T22:14:04.8726653Z OK 2023-01-11T22:14:04.8726671Z 2023-01-11T22:14:04.8726795Z Generating XML reports... 2023-01-11T22:14:04.8727247Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215024.xml 2023-01-11T22:14:04.8727613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8727770Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8728144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8728385Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8728405Z 2023-01-11T22:14:04.8728512Z Running tests... 2023-01-11T22:14:04.8728774Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8729083Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8729343Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8729559Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33378 2023-01-11T22:14:04.8729755Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33379 2023-01-11T22:14:04.8730123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8730504Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8730902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8731086Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8731444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8731687Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8732069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8732251Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8732475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8732713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8733114Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8733498Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8733723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8733947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8734102Z skip: Skipped due to small world size. (4.222s) 2023-01-11T22:14:04.8734122Z 2023-01-11T22:14:04.8734374Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8734482Z Ran 1 test in 4.222s 2023-01-11T22:14:04.8734502Z 2023-01-11T22:14:04.8734590Z OK (skipped=1) 2023-01-11T22:14:04.8734609Z 2023-01-11T22:14:04.8734731Z Generating XML reports... 2023-01-11T22:14:04.8735174Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215031.xml 2023-01-11T22:14:04.8735543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8735714Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8736083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8736273Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8736293Z 2023-01-11T22:14:04.8736398Z Running tests... 2023-01-11T22:14:04.8736641Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8736951Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8737205Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8737495Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33487 2023-01-11T22:14:04.8737704Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33488 2023-01-11T22:14:04.8738076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8738247Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8738624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8738810Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8739152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8739319Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8739689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8739878Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8740117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8740354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8740787Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8741184Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8741409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8741616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8741771Z skip: Skipped due to small world size. (4.151s) 2023-01-11T22:14:04.8741794Z 2023-01-11T22:14:04.8742053Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8742163Z Ran 1 test in 4.151s 2023-01-11T22:14:04.8742182Z 2023-01-11T22:14:04.8742281Z OK (skipped=1) 2023-01-11T22:14:04.8742300Z 2023-01-11T22:14:04.8742419Z Generating XML reports... 2023-01-11T22:14:04.8742867Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215037.xml 2023-01-11T22:14:04.8743237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8743393Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8743769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8743953Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8743976Z 2023-01-11T22:14:04.8744081Z Running tests... 2023-01-11T22:14:04.8744385Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8744697Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8744963Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8745182Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33596 2023-01-11T22:14:04.8745395Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33597 2023-01-11T22:14:04.8745745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8745916Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8746286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8746527Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8746886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8747056Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8747422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8747609Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8747836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8748077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8748471Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8748866Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8749089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8749311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8749465Z skip: Skipped due to small world size. (4.138s) 2023-01-11T22:14:04.8749485Z 2023-01-11T22:14:04.8749792Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8749909Z Ran 1 test in 4.138s 2023-01-11T22:14:04.8749928Z 2023-01-11T22:14:04.8750017Z OK (skipped=1) 2023-01-11T22:14:04.8750036Z 2023-01-11T22:14:04.8750158Z Generating XML reports... 2023-01-11T22:14:04.8750607Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215044.xml 2023-01-11T22:14:04.8750972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8751147Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8751519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8751707Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8751726Z 2023-01-11T22:14:04.8751833Z Running tests... 2023-01-11T22:14:04.8752095Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8752388Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8752647Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8752862Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33705 2023-01-11T22:14:04.8753075Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33706 2023-01-11T22:14:04.8753443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8753615Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8753989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8754174Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8754520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8754690Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8755057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8755238Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8755536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8755775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8756171Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8756564Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8756791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8756997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8757153Z skip: Skipped due to small world size. (4.271s) 2023-01-11T22:14:04.8757172Z 2023-01-11T22:14:04.8757435Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8757547Z Ran 1 test in 4.271s 2023-01-11T22:14:04.8757567Z 2023-01-11T22:14:04.8757671Z OK (skipped=1) 2023-01-11T22:14:04.8757689Z 2023-01-11T22:14:04.8757810Z Generating XML reports... 2023-01-11T22:14:04.8758256Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215051.xml 2023-01-11T22:14:04.8758621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8758825Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8759210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8759398Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8759418Z 2023-01-11T22:14:04.8759524Z Running tests... 2023-01-11T22:14:04.8759782Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8760094Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8760341Z test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8760554Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33814 2023-01-11T22:14:04.8760767Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33815 2023-01-11T22:14:04.8761118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8761289Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8761663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8761850Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8762209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8762380Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8762752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8762935Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8763161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8763401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8763796Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8764186Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8764411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8764689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8765023Z STAGE:2023-01-11 21:51:01 33815:33815 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8765341Z STAGE:2023-01-11 21:51:01 33814:33814 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8765674Z STAGE:2023-01-11 21:51:01 33814:33814 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8765987Z STAGE:2023-01-11 21:51:01 33815:33815 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8766329Z STAGE:2023-01-11 21:51:01 33814:33814 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8766668Z STAGE:2023-01-11 21:51:01 33815:33815 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8767188Z STAGE:2023-01-11 21:51:01 33814:33814 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:51:01 33815:33815 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8767211Z 2023-01-11T22:14:04.8767542Z STAGE:2023-01-11 21:51:01 33815:33815 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8767863Z STAGE:2023-01-11 21:51:01 33814:33814 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8768246Z STAGE:2023-01-11 21:51:01 33815:33815 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8768595Z STAGE:2023-01-11 21:51:01 33814:33814 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8768695Z ok (4.273s) 2023-01-11T22:14:04.8768715Z 2023-01-11T22:14:04.8768975Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8769069Z Ran 1 test in 4.273s 2023-01-11T22:14:04.8769088Z 2023-01-11T22:14:04.8769181Z OK 2023-01-11T22:14:04.8769203Z 2023-01-11T22:14:04.8769324Z Generating XML reports... 2023-01-11T22:14:04.8769770Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215058.xml 2023-01-11T22:14:04.8770138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8770534Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8770937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8771126Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8771148Z 2023-01-11T22:14:04.8771255Z Running tests... 2023-01-11T22:14:04.8771497Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8771805Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8772055Z test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8772270Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33927 2023-01-11T22:14:04.8772482Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33928 2023-01-11T22:14:04.8772847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8773021Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8773394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8773566Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8773924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8774093Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8774569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8774754Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8774993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8775237Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8775630Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8776020Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8776230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8776455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8776788Z STAGE:2023-01-11 21:51:08 33928:33928 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8777111Z STAGE:2023-01-11 21:51:08 33927:33927 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8777441Z STAGE:2023-01-11 21:51:08 33928:33928 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8777829Z STAGE:2023-01-11 21:51:08 33927:33927 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8778398Z STAGE:2023-01-11 21:51:08 33928:33928 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:51:08 33927:33927 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8778419Z 2023-01-11T22:14:04.8778743Z STAGE:2023-01-11 21:51:08 33927:33927 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8779064Z STAGE:2023-01-11 21:51:08 33928:33928 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8779374Z STAGE:2023-01-11 21:51:08 33928:33928 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8779696Z STAGE:2023-01-11 21:51:08 33927:33927 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8780035Z STAGE:2023-01-11 21:51:08 33928:33928 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8780378Z STAGE:2023-01-11 21:51:08 33927:33927 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8780478Z ok (4.255s) 2023-01-11T22:14:04.8780498Z 2023-01-11T22:14:04.8780758Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8780869Z Ran 1 test in 4.255s 2023-01-11T22:14:04.8780888Z 2023-01-11T22:14:04.8780980Z OK 2023-01-11T22:14:04.8780999Z 2023-01-11T22:14:04.8781121Z Generating XML reports... 2023-01-11T22:14:04.8781552Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215104.xml 2023-01-11T22:14:04.8781917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8782090Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8782463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8782655Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8782674Z 2023-01-11T22:14:04.8782781Z Running tests... 2023-01-11T22:14:04.8783037Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8783346Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8783584Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8783856Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34040 2023-01-11T22:14:04.8784070Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34041 2023-01-11T22:14:04.8784439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8784611Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8784985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8785174Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8785533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8785702Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8786052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8786242Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8786485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8786725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8787170Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8787571Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8787798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8788021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8788354Z STAGE:2023-01-11 21:51:17 34040:34040 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8789099Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1582: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2023-01-11T22:14:04.8789212Z warnings.warn( 2023-01-11T22:14:04.8789544Z STAGE:2023-01-11 21:51:17 34041:34041 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8790292Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1582: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2023-01-11T22:14:04.8790399Z warnings.warn( 2023-01-11T22:14:04.8790733Z STAGE:2023-01-11 21:51:17 34040:34040 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8791067Z STAGE:2023-01-11 21:51:17 34041:34041 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8791408Z STAGE:2023-01-11 21:51:17 34040:34040 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8791748Z STAGE:2023-01-11 21:51:17 34041:34041 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8792073Z STAGE:2023-01-11 21:51:17 34040:34040 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8792378Z STAGE:2023-01-11 21:51:17 34041:34041 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8792706Z STAGE:2023-01-11 21:51:17 34040:34040 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8793028Z STAGE:2023-01-11 21:51:17 34041:34041 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8793426Z STAGE:2023-01-11 21:51:17 34040:34040 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8793765Z STAGE:2023-01-11 21:51:17 34041:34041 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8793864Z ok (6.039s) 2023-01-11T22:14:04.8793885Z 2023-01-11T22:14:04.8794146Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8794260Z Ran 1 test in 6.039s 2023-01-11T22:14:04.8794280Z 2023-01-11T22:14:04.8794358Z OK 2023-01-11T22:14:04.8794395Z 2023-01-11T22:14:04.8794502Z Generating XML reports... 2023-01-11T22:14:04.8794950Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215111.xml 2023-01-11T22:14:04.8795313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8795484Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8795864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8796051Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8796070Z 2023-01-11T22:14:04.8796176Z Running tests... 2023-01-11T22:14:04.8796433Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8796771Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8797048Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8797261Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34155 2023-01-11T22:14:04.8797472Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34156 2023-01-11T22:14:04.8797839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8798013Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8798385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8798571Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8798913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8799086Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8799453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8799637Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8799878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8800117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8800516Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8800903Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8801127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8801335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8801665Z STAGE:2023-01-11 21:51:25 34156:34156 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8802417Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1582: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2023-01-11T22:14:04.8802592Z warnings.warn( 2023-01-11T22:14:04.8802922Z STAGE:2023-01-11 21:51:25 34155:34155 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8803677Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1582: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2023-01-11T22:14:04.8803787Z warnings.warn( 2023-01-11T22:14:04.8804118Z STAGE:2023-01-11 21:51:25 34155:34155 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8804446Z STAGE:2023-01-11 21:51:25 34156:34156 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8804786Z STAGE:2023-01-11 21:51:25 34155:34155 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8805116Z STAGE:2023-01-11 21:51:25 34156:34156 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8805633Z STAGE:2023-01-11 21:51:25 34155:34155 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:51:25 34156:34156 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8805653Z 2023-01-11T22:14:04.8805980Z STAGE:2023-01-11 21:51:25 34155:34155 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8806352Z STAGE:2023-01-11 21:51:25 34156:34156 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8806701Z STAGE:2023-01-11 21:51:25 34155:34155 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8807038Z STAGE:2023-01-11 21:51:25 34156:34156 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8807138Z ok (6.060s) 2023-01-11T22:14:04.8807158Z 2023-01-11T22:14:04.8807420Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8807534Z Ran 1 test in 6.060s 2023-01-11T22:14:04.8807554Z 2023-01-11T22:14:04.8807629Z OK 2023-01-11T22:14:04.8807664Z 2023-01-11T22:14:04.8807770Z Generating XML reports... 2023-01-11T22:14:04.8808217Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215120.xml 2023-01-11T22:14:04.8808587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8808760Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8809135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8809322Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8809341Z 2023-01-11T22:14:04.8809447Z Running tests... 2023-01-11T22:14:04.8809705Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8810001Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8810257Z test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8810697Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34270 2023-01-11T22:14:04.8810913Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34271 2023-01-11T22:14:04.8811287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8811458Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8811835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8812025Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8812473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8812627Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8812993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8813178Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8813420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8813659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8814053Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8814441Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8814670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8814877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8815208Z STAGE:2023-01-11 21:51:32 34271:34271 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8815523Z STAGE:2023-01-11 21:51:32 34270:34270 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8815914Z STAGE:2023-01-11 21:51:32 34270:34270 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8816255Z STAGE:2023-01-11 21:51:32 34271:34271 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8816599Z STAGE:2023-01-11 21:51:32 34270:34270 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8816938Z STAGE:2023-01-11 21:51:32 34271:34271 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8817461Z STAGE:2023-01-11 21:51:32 34270:34270 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:51:32 34271:34271 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8817483Z 2023-01-11T22:14:04.8817810Z STAGE:2023-01-11 21:51:32 34271:34271 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8818136Z STAGE:2023-01-11 21:51:32 34270:34270 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8818457Z STAGE:2023-01-11 21:51:32 34271:34271 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8818796Z STAGE:2023-01-11 21:51:32 34270:34270 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8818897Z ok (4.265s) 2023-01-11T22:14:04.8818916Z 2023-01-11T22:14:04.8819176Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8819287Z Ran 1 test in 4.265s 2023-01-11T22:14:04.8819310Z 2023-01-11T22:14:04.8819402Z OK 2023-01-11T22:14:04.8819421Z 2023-01-11T22:14:04.8819543Z Generating XML reports... 2023-01-11T22:14:04.8819989Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215128.xml 2023-01-11T22:14:04.8820356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8820516Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8820893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8821081Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8821100Z 2023-01-11T22:14:04.8821206Z Running tests... 2023-01-11T22:14:04.8821465Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8821772Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8822088Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8822303Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34383 2023-01-11T22:14:04.8822498Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34384 2023-01-11T22:14:04.8822870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8823041Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8823411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8823598Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8823957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8824128Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8824492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8824678Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8824902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8825188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8825595Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8825988Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8826211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8826440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8826540Z ok (5.867s) 2023-01-11T22:14:04.8826559Z 2023-01-11T22:14:04.8826822Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8826916Z Ran 1 test in 5.868s 2023-01-11T22:14:04.8826952Z 2023-01-11T22:14:04.8827027Z OK 2023-01-11T22:14:04.8827046Z 2023-01-11T22:14:04.8827168Z Generating XML reports... 2023-01-11T22:14:04.8827619Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215135.xml 2023-01-11T22:14:04.8827986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8828160Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8828536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8828727Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8828747Z 2023-01-11T22:14:04.8828855Z Running tests... 2023-01-11T22:14:04.8829101Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8829409Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8829660Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8829875Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34494 2023-01-11T22:14:04.8830089Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34495 2023-01-11T22:14:04.8830454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8830625Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8831059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8831231Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8831590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8831760Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8832134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8832317Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8832556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8832792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8833193Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8833581Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8833789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8834012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8834389Z STAGE:2023-01-11 21:51:47 34495:34495 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8834717Z STAGE:2023-01-11 21:51:47 34494:34494 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8835251Z STAGE:2023-01-11 21:51:47 34495:34495 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:51:47 34494:34494 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8835275Z 2023-01-11T22:14:04.8835835Z STAGE:2023-01-11 21:51:47 34495:34495 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:51:47 34494:34494 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8835855Z 2023-01-11T22:14:04.8836368Z STAGE:2023-01-11 21:51:47 34495:34495 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:51:47 34494:34494 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8836391Z 2023-01-11T22:14:04.8836722Z STAGE:2023-01-11 21:51:47 34495:34495 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8837045Z STAGE:2023-01-11 21:51:47 34494:34494 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8837384Z STAGE:2023-01-11 21:51:47 34495:34495 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8837727Z STAGE:2023-01-11 21:51:47 34494:34494 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8837815Z ok (4.215s) 2023-01-11T22:14:04.8837851Z 2023-01-11T22:14:04.8838095Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8838205Z Ran 1 test in 4.215s 2023-01-11T22:14:04.8838224Z 2023-01-11T22:14:04.8838314Z OK 2023-01-11T22:14:04.8838333Z 2023-01-11T22:14:04.8838455Z Generating XML reports... 2023-01-11T22:14:04.8838904Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215144.xml 2023-01-11T22:14:04.8839270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8839441Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8839818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8839990Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8840123Z 2023-01-11T22:14:04.8840236Z Running tests... 2023-01-11T22:14:04.8840497Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8840809Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8841066Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8841284Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34607 2023-01-11T22:14:04.8841498Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34608 2023-01-11T22:14:04.8841862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8842016Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8842388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8842576Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8842935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8843103Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8843514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8843708Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8843948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8844224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8844609Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8845004Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8845228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8845452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8845789Z STAGE:2023-01-11 21:51:54 34608:34608 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8846107Z STAGE:2023-01-11 21:51:54 34607:34607 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8846436Z STAGE:2023-01-11 21:51:54 34607:34607 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8846761Z STAGE:2023-01-11 21:51:54 34608:34608 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8847319Z STAGE:2023-01-11 21:51:54 34607:34607 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:51:54 34608:34608 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8847343Z 2023-01-11T22:14:04.8847671Z STAGE:2023-01-11 21:51:54 34608:34608 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8847970Z STAGE:2023-01-11 21:51:54 34607:34607 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8848302Z STAGE:2023-01-11 21:51:54 34608:34608 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8848625Z STAGE:2023-01-11 21:51:54 34607:34607 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8848966Z STAGE:2023-01-11 21:51:54 34608:34608 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8849304Z STAGE:2023-01-11 21:51:54 34607:34607 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8849405Z ok (4.238s) 2023-01-11T22:14:04.8849472Z 2023-01-11T22:14:04.8849742Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8849853Z Ran 1 test in 4.238s 2023-01-11T22:14:04.8849873Z 2023-01-11T22:14:04.8849948Z OK 2023-01-11T22:14:04.8849984Z 2023-01-11T22:14:04.8850089Z Generating XML reports... 2023-01-11T22:14:04.8850707Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215150.xml 2023-01-11T22:14:04.8851093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8851266Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8851640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8851827Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8851847Z 2023-01-11T22:14:04.8851956Z Running tests... 2023-01-11T22:14:04.8852214Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8852506Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8852765Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8853016Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34720 2023-01-11T22:14:04.8853311Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34721 2023-01-11T22:14:04.8853697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8853869Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8854243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8854430Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8854778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8854949Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8855317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8855505Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8855746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8855987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8856380Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8856769Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8856998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8857204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8857534Z STAGE:2023-01-11 21:52:01 34721:34721 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8857856Z STAGE:2023-01-11 21:52:01 34720:34720 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8858192Z STAGE:2023-01-11 21:52:01 34720:34720 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8858520Z STAGE:2023-01-11 21:52:01 34721:34721 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8858862Z STAGE:2023-01-11 21:52:01 34720:34720 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8859203Z STAGE:2023-01-11 21:52:01 34721:34721 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8859594Z STAGE:2023-01-11 21:52:01 34720:34720 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8859911Z STAGE:2023-01-11 21:52:01 34721:34721 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8860223Z STAGE:2023-01-11 21:52:01 34721:34721 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8860773Z STAGE:2023-01-11 21:52:01 34720:34720 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:52:01 34721:34721 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8860794Z 2023-01-11T22:14:04.8861132Z STAGE:2023-01-11 21:52:01 34720:34720 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8861233Z ok (4.216s) 2023-01-11T22:14:04.8861252Z 2023-01-11T22:14:04.8861513Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8861626Z Ran 1 test in 4.216s 2023-01-11T22:14:04.8861646Z 2023-01-11T22:14:04.8861738Z OK 2023-01-11T22:14:04.8861757Z 2023-01-11T22:14:04.8861879Z Generating XML reports... 2023-01-11T22:14:04.8862327Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215157.xml 2023-01-11T22:14:04.8862720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8862900Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8863276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8863464Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8863484Z 2023-01-11T22:14:04.8863588Z Running tests... 2023-01-11T22:14:04.8863847Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8864160Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8864417Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8864632Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34833 2023-01-11T22:14:04.8864828Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34834 2023-01-11T22:14:04.8865195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8865366Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8865742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8865929Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8866288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8866462Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8866833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8866999Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8867245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8867483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8867879Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8868269Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8868545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8868768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8869098Z STAGE:2023-01-11 21:52:09 34834:34834 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8869417Z STAGE:2023-01-11 21:52:09 34833:34833 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8869734Z STAGE:2023-01-11 21:52:09 34833:34833 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8870058Z STAGE:2023-01-11 21:52:09 34834:34834 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8870395Z STAGE:2023-01-11 21:52:09 34833:34833 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8870737Z STAGE:2023-01-11 21:52:09 34834:34834 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8871066Z STAGE:2023-01-11 21:52:09 34833:34833 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8871383Z STAGE:2023-01-11 21:52:09 34834:34834 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8871920Z STAGE:2023-01-11 21:52:11 34833:34833 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:52:11 34834:34834 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8871940Z 2023-01-11T22:14:04.8872541Z STAGE:2023-01-11 21:52:11 34833:34833 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:52:11 34834:34834 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8872563Z 2023-01-11T22:14:04.8872890Z STAGE:2023-01-11 21:52:11 34834:34834 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8873204Z STAGE:2023-01-11 21:52:11 34833:34833 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8873535Z STAGE:2023-01-11 21:52:11 34833:34833 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8873847Z STAGE:2023-01-11 21:52:11 34834:34834 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8874187Z STAGE:2023-01-11 21:52:11 34833:34833 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8874528Z STAGE:2023-01-11 21:52:11 34834:34834 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8874628Z ok (7.441s) 2023-01-11T22:14:04.8874647Z 2023-01-11T22:14:04.8874907Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8875017Z Ran 1 test in 7.441s 2023-01-11T22:14:04.8875036Z 2023-01-11T22:14:04.8875128Z OK 2023-01-11T22:14:04.8875147Z 2023-01-11T22:14:04.8875269Z Generating XML reports... 2023-01-11T22:14:04.8875698Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215204.xml 2023-01-11T22:14:04.8876072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8876244Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8876620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8876809Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8876829Z 2023-01-11T22:14:04.8876935Z Running tests... 2023-01-11T22:14:04.8877193Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8877504Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8877767Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8877965Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34949 2023-01-11T22:14:04.8878234Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34950 2023-01-11T22:14:04.8878602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8878772Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8879144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8879331Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8879688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8879857Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8880208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8880395Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8880636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8880876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8881319Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8881723Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8881947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8882167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8882499Z STAGE:2023-01-11 21:52:19 34950:34950 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8882806Z STAGE:2023-01-11 21:52:19 34949:34949 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8883136Z STAGE:2023-01-11 21:52:19 34949:34949 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8883456Z STAGE:2023-01-11 21:52:19 34950:34950 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8883799Z STAGE:2023-01-11 21:52:19 34949:34949 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8884140Z STAGE:2023-01-11 21:52:19 34950:34950 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8884460Z STAGE:2023-01-11 21:52:19 34950:34950 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8884776Z STAGE:2023-01-11 21:52:19 34949:34949 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8885103Z STAGE:2023-01-11 21:52:21 34949:34949 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8885430Z STAGE:2023-01-11 21:52:21 34950:34950 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8885990Z STAGE:2023-01-11 21:52:21 34949:34949 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:52:21 34950:34950 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8886012Z 2023-01-11T22:14:04.8886320Z STAGE:2023-01-11 21:52:21 34950:34950 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8886637Z STAGE:2023-01-11 21:52:21 34949:34949 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8886968Z STAGE:2023-01-11 21:52:21 34950:34950 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8887295Z STAGE:2023-01-11 21:52:21 34949:34949 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8887854Z STAGE:2023-01-11 21:52:21 34950:34950 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:52:21 34949:34949 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8887921Z 2023-01-11T22:14:04.8888026Z ok (7.321s) 2023-01-11T22:14:04.8888045Z 2023-01-11T22:14:04.8888307Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8888418Z Ran 1 test in 7.321s 2023-01-11T22:14:04.8888437Z 2023-01-11T22:14:04.8888529Z OK 2023-01-11T22:14:04.8888551Z 2023-01-11T22:14:04.8888673Z Generating XML reports... 2023-01-11T22:14:04.8889104Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215214.xml 2023-01-11T22:14:04.8889469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8889640Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8890017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8890208Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8890228Z 2023-01-11T22:14:04.8890625Z Running tests... 2023-01-11T22:14:04.8890904Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8891215Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8891544Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8891771Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35065 2023-01-11T22:14:04.8891984Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35066 2023-01-11T22:14:04.8892353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8892529Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8892892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8893063Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8893436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8893626Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8893983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8894166Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8894406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8894799Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8895037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8895428Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8895655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8895880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8896209Z STAGE:2023-01-11 21:52:29 35066:35066 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8896508Z STAGE:2023-01-11 21:52:29 35065:35065 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8896836Z STAGE:2023-01-11 21:52:29 35065:35065 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8897156Z STAGE:2023-01-11 21:52:29 35066:35066 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8897794Z STAGE:2023-01-11 21:52:29 35066:35066 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:52:29 35065:35065 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8897816Z 2023-01-11T22:14:04.8898140Z STAGE:2023-01-11 21:52:29 35066:35066 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8898460Z STAGE:2023-01-11 21:52:29 35065:35065 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8898994Z STAGE:2023-01-11 21:52:31 35066:35066 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:52:31 35065:35065 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8899014Z 2023-01-11T22:14:04.8899569Z STAGE:2023-01-11 21:52:31 35066:35066 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:52:31 35065:35065 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8899592Z 2023-01-11T22:14:04.8900108Z STAGE:2023-01-11 21:52:31 35066:35066 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:52:31 35065:35065 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.8900128Z 2023-01-11T22:14:04.8900451Z STAGE:2023-01-11 21:52:31 35065:35065 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8900818Z STAGE:2023-01-11 21:52:31 35066:35066 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.8901168Z STAGE:2023-01-11 21:52:31 35065:35065 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8901493Z STAGE:2023-01-11 21:52:31 35066:35066 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.8901594Z ok (7.451s) 2023-01-11T22:14:04.8901613Z 2023-01-11T22:14:04.8901873Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8901988Z Ran 1 test in 7.451s 2023-01-11T22:14:04.8902008Z 2023-01-11T22:14:04.8902099Z OK 2023-01-11T22:14:04.8902118Z 2023-01-11T22:14:04.8902239Z Generating XML reports... 2023-01-11T22:14:04.8902685Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215224.xml 2023-01-11T22:14:04.8903055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8903211Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8903589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8903776Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8903795Z 2023-01-11T22:14:04.8903901Z Running tests... 2023-01-11T22:14:04.8904160Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8904476Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8904718Z test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s) 2023-01-11T22:14:04.8904737Z 2023-01-11T22:14:04.8904996Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8905108Z Ran 1 test in 0.002s 2023-01-11T22:14:04.8905127Z 2023-01-11T22:14:04.8905219Z OK (skipped=1) 2023-01-11T22:14:04.8905238Z 2023-01-11T22:14:04.8905359Z Generating XML reports... 2023-01-11T22:14:04.8905802Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215234.xml 2023-01-11T22:14:04.8906166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8906338Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8906781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8906967Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8906987Z 2023-01-11T22:14:04.8907092Z Running tests... 2023-01-11T22:14:04.8907333Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8907645Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8907897Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s) 2023-01-11T22:14:04.8907916Z 2023-01-11T22:14:04.8908174Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8908284Z Ran 1 test in 0.002s 2023-01-11T22:14:04.8908304Z 2023-01-11T22:14:04.8908409Z OK (skipped=1) 2023-01-11T22:14:04.8908427Z 2023-01-11T22:14:04.8908548Z Generating XML reports... 2023-01-11T22:14:04.8908991Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215236.xml 2023-01-11T22:14:04.8909356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8909512Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8909933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8910130Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8910148Z 2023-01-11T22:14:04.8910256Z Running tests... 2023-01-11T22:14:04.8910517Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8910826Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8911080Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s) 2023-01-11T22:14:04.8911104Z 2023-01-11T22:14:04.8911358Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8911467Z Ran 1 test in 0.002s 2023-01-11T22:14:04.8911487Z 2023-01-11T22:14:04.8911575Z OK (skipped=1) 2023-01-11T22:14:04.8911594Z 2023-01-11T22:14:04.8911715Z Generating XML reports... 2023-01-11T22:14:04.8912159Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215239.xml 2023-01-11T22:14:04.8912525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8912694Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8913067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8913255Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8913277Z 2023-01-11T22:14:04.8913384Z Running tests... 2023-01-11T22:14:04.8913645Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8913936Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8914198Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s) 2023-01-11T22:14:04.8914217Z 2023-01-11T22:14:04.8914476Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8914585Z Ran 1 test in 0.002s 2023-01-11T22:14:04.8914605Z 2023-01-11T22:14:04.8914709Z OK (skipped=1) 2023-01-11T22:14:04.8914728Z 2023-01-11T22:14:04.8914849Z Generating XML reports... 2023-01-11T22:14:04.8915290Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215241.xml 2023-01-11T22:14:04.8915651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8915859Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8916237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8916425Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8916444Z 2023-01-11T22:14:04.8916551Z Running tests... 2023-01-11T22:14:04.8916812Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8917122Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8917376Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s) 2023-01-11T22:14:04.8917395Z 2023-01-11T22:14:04.8917649Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8917763Z Ran 1 test in 0.002s 2023-01-11T22:14:04.8917782Z 2023-01-11T22:14:04.8917871Z OK (skipped=1) 2023-01-11T22:14:04.8917906Z 2023-01-11T22:14:04.8918011Z Generating XML reports... 2023-01-11T22:14:04.8918454Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215243.xml 2023-01-11T22:14:04.8918817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8919034Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8919417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8919604Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8919623Z 2023-01-11T22:14:04.8919731Z Running tests... 2023-01-11T22:14:04.8919990Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8920286Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8920553Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s) 2023-01-11T22:14:04.8920572Z 2023-01-11T22:14:04.8920828Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8920936Z Ran 1 test in 0.002s 2023-01-11T22:14:04.8920955Z 2023-01-11T22:14:04.8921061Z OK (skipped=1) 2023-01-11T22:14:04.8921083Z 2023-01-11T22:14:04.8921204Z Generating XML reports... 2023-01-11T22:14:04.8921648Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215246.xml 2023-01-11T22:14:04.8922011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8922187Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8922545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8922731Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8922750Z 2023-01-11T22:14:04.8922856Z Running tests... 2023-01-11T22:14:04.8923114Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8923423Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8923674Z test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s) 2023-01-11T22:14:04.8923694Z 2023-01-11T22:14:04.8923952Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8924060Z Ran 1 test in 0.002s 2023-01-11T22:14:04.8924080Z 2023-01-11T22:14:04.8924170Z OK (skipped=1) 2023-01-11T22:14:04.8924205Z 2023-01-11T22:14:04.8924309Z Generating XML reports... 2023-01-11T22:14:04.8924820Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215248.xml 2023-01-11T22:14:04.8925186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8925356Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8925728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8925918Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8925938Z 2023-01-11T22:14:04.8926044Z Running tests... 2023-01-11T22:14:04.8926303Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8926592Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8926863Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2023-01-11T22:14:04.8926886Z 2023-01-11T22:14:04.8927141Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8927250Z Ran 1 test in 0.002s 2023-01-11T22:14:04.8927270Z 2023-01-11T22:14:04.8927374Z OK (skipped=1) 2023-01-11T22:14:04.8927392Z 2023-01-11T22:14:04.8927513Z Generating XML reports... 2023-01-11T22:14:04.8927999Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215250.xml 2023-01-11T22:14:04.8928373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8928544Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8928898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8929084Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8929106Z 2023-01-11T22:14:04.8929211Z Running tests... 2023-01-11T22:14:04.8929471Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8929776Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8930052Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s) 2023-01-11T22:14:04.8930071Z 2023-01-11T22:14:04.8930555Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8930675Z Ran 1 test in 0.002s 2023-01-11T22:14:04.8930695Z 2023-01-11T22:14:04.8930802Z OK (skipped=1) 2023-01-11T22:14:04.8930822Z 2023-01-11T22:14:04.8930926Z Generating XML reports... 2023-01-11T22:14:04.8931370Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215253.xml 2023-01-11T22:14:04.8931730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8931905Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8932275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8932459Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8932478Z 2023-01-11T22:14:04.8932583Z Running tests... 2023-01-11T22:14:04.8932845Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8933134Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8933422Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s) 2023-01-11T22:14:04.8933442Z 2023-01-11T22:14:04.8933699Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8933891Z Ran 1 test in 0.002s 2023-01-11T22:14:04.8933910Z 2023-01-11T22:14:04.8934015Z OK (skipped=1) 2023-01-11T22:14:04.8934033Z 2023-01-11T22:14:04.8934155Z Generating XML reports... 2023-01-11T22:14:04.8934600Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215255.xml 2023-01-11T22:14:04.8934961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8935138Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8935494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8935682Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8935701Z 2023-01-11T22:14:04.8935806Z Running tests... 2023-01-11T22:14:04.8936062Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8936375Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8936660Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2023-01-11T22:14:04.8936680Z 2023-01-11T22:14:04.8936937Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8937046Z Ran 1 test in 0.002s 2023-01-11T22:14:04.8937065Z 2023-01-11T22:14:04.8937232Z OK (skipped=1) 2023-01-11T22:14:04.8937253Z 2023-01-11T22:14:04.8937365Z Generating XML reports... 2023-01-11T22:14:04.8937809Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215258.xml 2023-01-11T22:14:04.8938171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8938341Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8938717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8938904Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8938923Z 2023-01-11T22:14:04.8939028Z Running tests... 2023-01-11T22:14:04.8939287Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8939597Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8939881Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2023-01-11T22:14:04.8939901Z 2023-01-11T22:14:04.8940158Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8940266Z Ran 1 test in 0.002s 2023-01-11T22:14:04.8940286Z 2023-01-11T22:14:04.8940390Z OK (skipped=1) 2023-01-11T22:14:04.8940409Z 2023-01-11T22:14:04.8940533Z Generating XML reports... 2023-01-11T22:14:04.8940971Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215300.xml 2023-01-11T22:14:04.8941333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8941503Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8941879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8942049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8942068Z 2023-01-11T22:14:04.8942174Z Running tests... 2023-01-11T22:14:04.8942433Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8942738Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8943027Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s) 2023-01-11T22:14:04.8943097Z 2023-01-11T22:14:04.8943359Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8943469Z Ran 1 test in 0.002s 2023-01-11T22:14:04.8943488Z 2023-01-11T22:14:04.8943593Z OK (skipped=1) 2023-01-11T22:14:04.8943612Z 2023-01-11T22:14:04.8943717Z Generating XML reports... 2023-01-11T22:14:04.8944164Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215302.xml 2023-01-11T22:14:04.8944576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8944748Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8945123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8945315Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8945334Z 2023-01-11T22:14:04.8945439Z Running tests... 2023-01-11T22:14:04.8945700Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8946007Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8946342Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2023-01-11T22:14:04.8946381Z 2023-01-11T22:14:04.8946629Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8946738Z Ran 1 test in 0.002s 2023-01-11T22:14:04.8946757Z 2023-01-11T22:14:04.8946862Z OK (skipped=1) 2023-01-11T22:14:04.8946881Z 2023-01-11T22:14:04.8947001Z Generating XML reports... 2023-01-11T22:14:04.8947442Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215305.xml 2023-01-11T22:14:04.8947810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8947980Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8948352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8948519Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8948541Z 2023-01-11T22:14:04.8948650Z Running tests... 2023-01-11T22:14:04.8948907Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8949214Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8949500Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s) 2023-01-11T22:14:04.8949520Z 2023-01-11T22:14:04.8949781Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8949889Z Ran 1 test in 0.002s 2023-01-11T22:14:04.8949908Z 2023-01-11T22:14:04.8950012Z OK (skipped=1) 2023-01-11T22:14:04.8950030Z 2023-01-11T22:14:04.8950153Z Generating XML reports... 2023-01-11T22:14:04.8950575Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215307.xml 2023-01-11T22:14:04.8950943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8951115Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8951488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8951673Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8951692Z 2023-01-11T22:14:04.8951796Z Running tests... 2023-01-11T22:14:04.8952113Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8952421Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8952696Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2023-01-11T22:14:04.8952733Z 2023-01-11T22:14:04.8952973Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8953088Z Ran 1 test in 0.002s 2023-01-11T22:14:04.8953107Z 2023-01-11T22:14:04.8953212Z OK (skipped=1) 2023-01-11T22:14:04.8953231Z 2023-01-11T22:14:04.8953350Z Generating XML reports... 2023-01-11T22:14:04.8953789Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215310.xml 2023-01-11T22:14:04.8954153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8954328Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8954702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8954873Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8954909Z 2023-01-11T22:14:04.8954998Z Running tests... 2023-01-11T22:14:04.8955304Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8955625Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8955902Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s) 2023-01-11T22:14:04.8955922Z 2023-01-11T22:14:04.8956177Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8956285Z Ran 1 test in 0.002s 2023-01-11T22:14:04.8956304Z 2023-01-11T22:14:04.8956412Z OK (skipped=1) 2023-01-11T22:14:04.8956431Z 2023-01-11T22:14:04.8956550Z Generating XML reports... 2023-01-11T22:14:04.8956970Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215312.xml 2023-01-11T22:14:04.8957333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8957504Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8957878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8958064Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8958083Z 2023-01-11T22:14:04.8958187Z Running tests... 2023-01-11T22:14:04.8958444Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8958750Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8959046Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s) 2023-01-11T22:14:04.8959066Z 2023-01-11T22:14:04.8959305Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8959416Z Ran 1 test in 0.002s 2023-01-11T22:14:04.8959435Z 2023-01-11T22:14:04.8959544Z OK (skipped=1) 2023-01-11T22:14:04.8959563Z 2023-01-11T22:14:04.8959686Z Generating XML reports... 2023-01-11T22:14:04.8960134Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215314.xml 2023-01-11T22:14:04.8960499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8960672Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8961047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8961289Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8961308Z 2023-01-11T22:14:04.8961398Z Running tests... 2023-01-11T22:14:04.8961660Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8961967Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8962260Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2023-01-11T22:14:04.8962280Z 2023-01-11T22:14:04.8962541Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8962650Z Ran 1 test in 0.002s 2023-01-11T22:14:04.8962669Z 2023-01-11T22:14:04.8962775Z OK (skipped=1) 2023-01-11T22:14:04.8962793Z 2023-01-11T22:14:04.8962912Z Generating XML reports... 2023-01-11T22:14:04.8963351Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215317.xml 2023-01-11T22:14:04.8963702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8963874Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8964246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8964479Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8964499Z 2023-01-11T22:14:04.8964610Z Running tests... 2023-01-11T22:14:04.8964870Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8965177Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8965476Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2023-01-11T22:14:04.8965499Z 2023-01-11T22:14:04.8965759Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8965851Z Ran 1 test in 0.002s 2023-01-11T22:14:04.8965870Z 2023-01-11T22:14:04.8965975Z OK (skipped=1) 2023-01-11T22:14:04.8965993Z 2023-01-11T22:14:04.8966113Z Generating XML reports... 2023-01-11T22:14:04.8966554Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215319.xml 2023-01-11T22:14:04.8966915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8967086Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8967453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8967639Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8967664Z 2023-01-11T22:14:04.8967753Z Running tests... 2023-01-11T22:14:04.8968013Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8968322Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8968618Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s) 2023-01-11T22:14:04.8968637Z 2023-01-11T22:14:04.8968895Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8969004Z Ran 1 test in 0.002s 2023-01-11T22:14:04.8969023Z 2023-01-11T22:14:04.8969126Z OK (skipped=1) 2023-01-11T22:14:04.8969145Z 2023-01-11T22:14:04.8969266Z Generating XML reports... 2023-01-11T22:14:04.8969707Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215321.xml 2023-01-11T22:14:04.8970056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8970511Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8970909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8971096Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8971117Z 2023-01-11T22:14:04.8971223Z Running tests... 2023-01-11T22:14:04.8971487Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8971797Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8972100Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2023-01-11T22:14:04.8972120Z 2023-01-11T22:14:04.8972377Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8972474Z Ran 1 test in 0.002s 2023-01-11T22:14:04.8972493Z 2023-01-11T22:14:04.8972598Z OK (skipped=1) 2023-01-11T22:14:04.8972617Z 2023-01-11T22:14:04.8972737Z Generating XML reports... 2023-01-11T22:14:04.8973176Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215324.xml 2023-01-11T22:14:04.8973615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8973796Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8974172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8974359Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8974379Z 2023-01-11T22:14:04.8974485Z Running tests... 2023-01-11T22:14:04.8974728Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8975040Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8975331Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s) 2023-01-11T22:14:04.8975350Z 2023-01-11T22:14:04.8975606Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8975714Z Ran 1 test in 0.002s 2023-01-11T22:14:04.8975733Z 2023-01-11T22:14:04.8975841Z OK (skipped=1) 2023-01-11T22:14:04.8975860Z 2023-01-11T22:14:04.8975979Z Generating XML reports... 2023-01-11T22:14:04.8976424Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215326.xml 2023-01-11T22:14:04.8976772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8976945Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8977324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8977510Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8977529Z 2023-01-11T22:14:04.8977633Z Running tests... 2023-01-11T22:14:04.8977891Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8978200Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8978498Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2023-01-11T22:14:04.8978520Z 2023-01-11T22:14:04.8978774Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8978866Z Ran 1 test in 0.002s 2023-01-11T22:14:04.8978901Z 2023-01-11T22:14:04.8978990Z OK (skipped=1) 2023-01-11T22:14:04.8979076Z 2023-01-11T22:14:04.8979206Z Generating XML reports... 2023-01-11T22:14:04.8979647Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215329.xml 2023-01-11T22:14:04.8980011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8980182Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8980557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8980745Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8980764Z 2023-01-11T22:14:04.8980870Z Running tests... 2023-01-11T22:14:04.8981111Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8981416Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8981678Z test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8981893Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35973 2023-01-11T22:14:04.8982106Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35974 2023-01-11T22:14:04.8982476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8982741Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8983128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8983297Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8983656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8983824Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8984201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8984386Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8984630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8984872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8985268Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8985660Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8985868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8986090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8986327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T22:14:04.8986563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T22:14:04.8986955Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.8987343Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.8987445Z ok (6.982s) 2023-01-11T22:14:04.8987464Z 2023-01-11T22:14:04.8987726Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8987835Z Ran 1 test in 6.982s 2023-01-11T22:14:04.8987855Z 2023-01-11T22:14:04.8987929Z OK 2023-01-11T22:14:04.8987949Z 2023-01-11T22:14:04.8988072Z Generating XML reports... 2023-01-11T22:14:04.8988580Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215331.xml 2023-01-11T22:14:04.8988944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8989114Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8989487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8989678Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8989697Z 2023-01-11T22:14:04.8989804Z Running tests... 2023-01-11T22:14:04.8990046Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8990355Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8990608Z test_backend_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.8990827Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36094 2023-01-11T22:14:04.8991041Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36095 2023-01-11T22:14:04.8991404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8991576Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8991999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8992193Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8992537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8992708Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8993082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8993270Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8993510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.8993750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.8994148Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8994537Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.8994762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.8994969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.8995122Z skip: Need at least 3 CUDA devices (4.248s) 2023-01-11T22:14:04.8995142Z 2023-01-11T22:14:04.8995404Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8995514Z Ran 1 test in 4.249s 2023-01-11T22:14:04.8995534Z 2023-01-11T22:14:04.8995639Z OK (skipped=1) 2023-01-11T22:14:04.8995658Z 2023-01-11T22:14:04.8995780Z Generating XML reports... 2023-01-11T22:14:04.8996226Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215341.xml 2023-01-11T22:14:04.8996590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8996746Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.8997121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.8997310Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.8997377Z 2023-01-11T22:14:04.8997487Z Running tests... 2023-01-11T22:14:04.8997752Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8998060Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.8998306Z test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 3 (0.002s) 2023-01-11T22:14:04.8998326Z 2023-01-11T22:14:04.8998585Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.8998696Z Ran 1 test in 0.002s 2023-01-11T22:14:04.8998715Z 2023-01-11T22:14:04.8998803Z OK (skipped=1) 2023-01-11T22:14:04.8998822Z 2023-01-11T22:14:04.8998944Z Generating XML reports... 2023-01-11T22:14:04.8999387Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215347.xml 2023-01-11T22:14:04.8999751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.8999926Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9000300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9000485Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9000505Z 2023-01-11T22:14:04.9000611Z Running tests... 2023-01-11T22:14:04.9000914Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9001216Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9001452Z test_barrier (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9001664Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36236 2023-01-11T22:14:04.9001877Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36237 2023-01-11T22:14:04.9002247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9002419Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9002785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9002970Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9003315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9003488Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9003855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9004037Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9004276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9004519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9004913Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9005303Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9005530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9005737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9005839Z ok (5.007s) 2023-01-11T22:14:04.9005859Z 2023-01-11T22:14:04.9006120Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9006230Z Ran 1 test in 5.007s 2023-01-11T22:14:04.9006249Z 2023-01-11T22:14:04.9006393Z OK 2023-01-11T22:14:04.9006412Z 2023-01-11T22:14:04.9006535Z Generating XML reports... 2023-01-11T22:14:04.9006982Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215350.xml 2023-01-11T22:14:04.9007350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9007506Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9007888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9008075Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9008094Z 2023-01-11T22:14:04.9008202Z Running tests... 2023-01-11T22:14:04.9008464Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9008772Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9009024Z test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9009239Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36345 2023-01-11T22:14:04.9009449Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36346 2023-01-11T22:14:04.9009799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9010022Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9010641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9010837Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9011203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9011380Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9011749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9011935Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9012160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9012402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9012800Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9013190Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9013414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9013642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9013743Z ok (6.515s) 2023-01-11T22:14:04.9013764Z 2023-01-11T22:14:04.9014025Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9014136Z Ran 1 test in 6.516s 2023-01-11T22:14:04.9014155Z 2023-01-11T22:14:04.9014230Z OK 2023-01-11T22:14:04.9014248Z 2023-01-11T22:14:04.9014371Z Generating XML reports... 2023-01-11T22:14:04.9014824Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215357.xml 2023-01-11T22:14:04.9015191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9015364Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9015734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9016084Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9016105Z 2023-01-11T22:14:04.9016212Z Running tests... 2023-01-11T22:14:04.9016459Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9016768Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9017025Z test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9017243Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36456 2023-01-11T22:14:04.9017456Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36457 2023-01-11T22:14:04.9017819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9017992Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9018368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9018558Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9018898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9019068Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9019499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9019696Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9019938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9020178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9020576Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9020971Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9021196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9021402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9021639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T22:14:04.9021877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T22:14:04.9022266Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.9022654Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.9022759Z ok (5.074s) 2023-01-11T22:14:04.9022779Z 2023-01-11T22:14:04.9023039Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9023150Z Ran 1 test in 5.075s 2023-01-11T22:14:04.9023170Z 2023-01-11T22:14:04.9023244Z OK 2023-01-11T22:14:04.9023280Z 2023-01-11T22:14:04.9023385Z Generating XML reports... 2023-01-11T22:14:04.9023832Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215406.xml 2023-01-11T22:14:04.9024202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9024377Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9024755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9024942Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9025013Z 2023-01-11T22:14:04.9025125Z Running tests... 2023-01-11T22:14:04.9025385Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9025677Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9025937Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9026155Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36571 2023-01-11T22:14:04.9026368Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36572 2023-01-11T22:14:04.9026733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9026905Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9027275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9027465Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9027825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9027977Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9028353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9028585Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9028833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9029072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9029469Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9029865Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9030090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9030297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9030453Z skip: Skipped due to small world size. (4.234s) 2023-01-11T22:14:04.9030473Z 2023-01-11T22:14:04.9030738Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9030849Z Ran 1 test in 4.234s 2023-01-11T22:14:04.9030868Z 2023-01-11T22:14:04.9030973Z OK (skipped=1) 2023-01-11T22:14:04.9030992Z 2023-01-11T22:14:04.9031115Z Generating XML reports... 2023-01-11T22:14:04.9031560Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215414.xml 2023-01-11T22:14:04.9031925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9032101Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9032457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9032643Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9032662Z 2023-01-11T22:14:04.9032768Z Running tests... 2023-01-11T22:14:04.9033032Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9033341Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9033588Z test_barrier_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9033802Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36680 2023-01-11T22:14:04.9034014Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36681 2023-01-11T22:14:04.9034436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9034610Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9034984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9035170Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9035533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9035703Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9036069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9036251Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9036496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9036872Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9037109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9037541Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9037773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9037997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9038152Z skip: Skipped due to small world size. (4.263s) 2023-01-11T22:14:04.9038171Z 2023-01-11T22:14:04.9038435Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9038549Z Ran 1 test in 4.263s 2023-01-11T22:14:04.9038569Z 2023-01-11T22:14:04.9038675Z OK (skipped=1) 2023-01-11T22:14:04.9038694Z 2023-01-11T22:14:04.9038797Z Generating XML reports... 2023-01-11T22:14:04.9039242Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215421.xml 2023-01-11T22:14:04.9039605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9039780Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9040154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9040342Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9040361Z 2023-01-11T22:14:04.9040467Z Running tests... 2023-01-11T22:14:04.9040732Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9041027Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9041283Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9041496Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36789 2023-01-11T22:14:04.9041708Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36790 2023-01-11T22:14:04.9042077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9042249Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9042619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9042804Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9043161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9043377Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9043750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9043933Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9044177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9044613Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9044845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9045238Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9045462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9045686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9045825Z skip: Skipped due to small world size. (4.202s) 2023-01-11T22:14:04.9045845Z 2023-01-11T22:14:04.9046105Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9046216Z Ran 1 test in 4.203s 2023-01-11T22:14:04.9046235Z 2023-01-11T22:14:04.9046339Z OK (skipped=1) 2023-01-11T22:14:04.9046407Z 2023-01-11T22:14:04.9046536Z Generating XML reports... 2023-01-11T22:14:04.9046987Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215428.xml 2023-01-11T22:14:04.9047354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9047525Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9047885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9048072Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9048091Z 2023-01-11T22:14:04.9048197Z Running tests... 2023-01-11T22:14:04.9048457Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9048765Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9049031Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9049246Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36898 2023-01-11T22:14:04.9049457Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36899 2023-01-11T22:14:04.9049822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9049981Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9050570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9050767Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9051129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9051304Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9051677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9051863Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9052103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9052323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9052805Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9053197Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9053420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9053648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9053883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T22:14:04.9054118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T22:14:04.9054510Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.9054902Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.9054986Z ok (5.326s) 2023-01-11T22:14:04.9055023Z 2023-01-11T22:14:04.9055268Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9055378Z Ran 1 test in 5.327s 2023-01-11T22:14:04.9055397Z 2023-01-11T22:14:04.9055489Z OK 2023-01-11T22:14:04.9055508Z 2023-01-11T22:14:04.9055629Z Generating XML reports... 2023-01-11T22:14:04.9056132Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215434.xml 2023-01-11T22:14:04.9056515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9056686Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9057066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9057241Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9057262Z 2023-01-11T22:14:04.9057370Z Running tests... 2023-01-11T22:14:04.9057629Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9057936Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9058200Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9058415Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37013 2023-01-11T22:14:04.9058627Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37014 2023-01-11T22:14:04.9058994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9059148Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9059528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9059716Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9060074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9060240Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9060613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9060798Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9061038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9061276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9061713Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9062103Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9062326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9062551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9062789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9063027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9063421Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9063808Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9063912Z ok (5.359s) 2023-01-11T22:14:04.9063932Z 2023-01-11T22:14:04.9064177Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9064289Z Ran 1 test in 5.359s 2023-01-11T22:14:04.9064308Z 2023-01-11T22:14:04.9064399Z OK 2023-01-11T22:14:04.9064418Z 2023-01-11T22:14:04.9064540Z Generating XML reports... 2023-01-11T22:14:04.9065032Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215442.xml 2023-01-11T22:14:04.9065409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9065581Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9065954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9066141Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9066165Z 2023-01-11T22:14:04.9066254Z Running tests... 2023-01-11T22:14:04.9066514Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9066824Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9067082Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9067299Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37128 2023-01-11T22:14:04.9067511Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37129 2023-01-11T22:14:04.9067876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9068047Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9068407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9068598Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9068959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9069127Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9069501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9069685Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9069925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9070162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9070554Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9070984Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9071209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9071432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9071589Z skip: Skipped due to small world size. (4.254s) 2023-01-11T22:14:04.9071612Z 2023-01-11T22:14:04.9071876Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9071987Z Ran 1 test in 4.255s 2023-01-11T22:14:04.9072006Z 2023-01-11T22:14:04.9072112Z OK (skipped=1) 2023-01-11T22:14:04.9072131Z 2023-01-11T22:14:04.9072253Z Generating XML reports... 2023-01-11T22:14:04.9072682Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215450.xml 2023-01-11T22:14:04.9073051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9073223Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9073599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9073787Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9073806Z 2023-01-11T22:14:04.9073966Z Running tests... 2023-01-11T22:14:04.9074241Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9074547Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9074809Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9075006Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37237 2023-01-11T22:14:04.9075224Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37238 2023-01-11T22:14:04.9075590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9075760Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9076133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9076324Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9076682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9076854Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9077205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9077387Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9077629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9077866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9078262Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9078655Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9078880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9079103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9079203Z ok (4.258s) 2023-01-11T22:14:04.9079223Z 2023-01-11T22:14:04.9079473Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9079637Z Ran 1 test in 4.258s 2023-01-11T22:14:04.9079656Z 2023-01-11T22:14:04.9079747Z OK 2023-01-11T22:14:04.9079766Z 2023-01-11T22:14:04.9079888Z Generating XML reports... 2023-01-11T22:14:04.9080333Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215457.xml 2023-01-11T22:14:04.9080699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9080874Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9081247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9081433Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9081453Z 2023-01-11T22:14:04.9081543Z Running tests... 2023-01-11T22:14:04.9081803Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9082118Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9082384Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9082598Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37346 2023-01-11T22:14:04.9082809Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37347 2023-01-11T22:14:04.9083220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9083399Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9083759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9083946Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9084300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9084475Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9084846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9085029Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9085272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9085512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9085907Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9086279Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9086508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9086731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9086831Z ok (4.227s) 2023-01-11T22:14:04.9086850Z 2023-01-11T22:14:04.9087112Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9087223Z Ran 1 test in 4.228s 2023-01-11T22:14:04.9087242Z 2023-01-11T22:14:04.9087334Z OK 2023-01-11T22:14:04.9087353Z 2023-01-11T22:14:04.9087478Z Generating XML reports... 2023-01-11T22:14:04.9087906Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215504.xml 2023-01-11T22:14:04.9088270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9088442Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9088815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9089058Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9089077Z 2023-01-11T22:14:04.9089183Z Running tests... 2023-01-11T22:14:04.9089443Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9089751Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9090025Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2023-01-11T22:14:04.9090046Z 2023-01-11T22:14:04.9090449Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9090572Z Ran 1 test in 0.002s 2023-01-11T22:14:04.9090592Z 2023-01-11T22:14:04.9090700Z OK (skipped=1) 2023-01-11T22:14:04.9090719Z 2023-01-11T22:14:04.9090842Z Generating XML reports... 2023-01-11T22:14:04.9091293Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215511.xml 2023-01-11T22:14:04.9091665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9091835Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9092206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9092465Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9092486Z 2023-01-11T22:14:04.9092582Z Running tests... 2023-01-11T22:14:04.9092843Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9093155Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9093410Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s) 2023-01-11T22:14:04.9093433Z 2023-01-11T22:14:04.9093689Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9093798Z Ran 1 test in 0.003s 2023-01-11T22:14:04.9093817Z 2023-01-11T22:14:04.9093921Z OK (skipped=1) 2023-01-11T22:14:04.9093940Z 2023-01-11T22:14:04.9094062Z Generating XML reports... 2023-01-11T22:14:04.9094487Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215513.xml 2023-01-11T22:14:04.9094855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9095026Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9095401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9095588Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9095611Z 2023-01-11T22:14:04.9095717Z Running tests... 2023-01-11T22:14:04.9095977Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9096285Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9096553Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2023-01-11T22:14:04.9096572Z 2023-01-11T22:14:04.9096817Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9096929Z Ran 1 test in 0.003s 2023-01-11T22:14:04.9096948Z 2023-01-11T22:14:04.9097054Z OK (skipped=1) 2023-01-11T22:14:04.9097072Z 2023-01-11T22:14:04.9097193Z Generating XML reports... 2023-01-11T22:14:04.9097639Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215515.xml 2023-01-11T22:14:04.9098004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9098262Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9098639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9098827Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9098847Z 2023-01-11T22:14:04.9098938Z Running tests... 2023-01-11T22:14:04.9099200Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9099507Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9099760Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2023-01-11T22:14:04.9099779Z 2023-01-11T22:14:04.9100032Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9100141Z Ran 1 test in 0.002s 2023-01-11T22:14:04.9100164Z 2023-01-11T22:14:04.9100270Z OK (skipped=1) 2023-01-11T22:14:04.9100289Z 2023-01-11T22:14:04.9100411Z Generating XML reports... 2023-01-11T22:14:04.9100854Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215518.xml 2023-01-11T22:14:04.9101203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9101432Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9101818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9102006Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9102026Z 2023-01-11T22:14:04.9102133Z Running tests... 2023-01-11T22:14:04.9102390Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9102698Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9102960Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2023-01-11T22:14:04.9102979Z 2023-01-11T22:14:04.9103238Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9103333Z Ran 1 test in 0.002s 2023-01-11T22:14:04.9103353Z 2023-01-11T22:14:04.9103458Z OK (skipped=1) 2023-01-11T22:14:04.9103477Z 2023-01-11T22:14:04.9103601Z Generating XML reports... 2023-01-11T22:14:04.9104040Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215520.xml 2023-01-11T22:14:04.9104402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9104573Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9104942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9105133Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9105152Z 2023-01-11T22:14:04.9105241Z Running tests... 2023-01-11T22:14:04.9105497Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9105803Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9106076Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2023-01-11T22:14:04.9106095Z 2023-01-11T22:14:04.9106353Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9106461Z Ran 1 test in 0.002s 2023-01-11T22:14:04.9106480Z 2023-01-11T22:14:04.9106585Z OK (skipped=1) 2023-01-11T22:14:04.9106604Z 2023-01-11T22:14:04.9106725Z Generating XML reports... 2023-01-11T22:14:04.9107169Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215523.xml 2023-01-11T22:14:04.9107574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9107748Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9108125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9108314Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9108334Z 2023-01-11T22:14:04.9108440Z Running tests... 2023-01-11T22:14:04.9108698Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9109004Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9109263Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2023-01-11T22:14:04.9109286Z 2023-01-11T22:14:04.9109544Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9109636Z Ran 1 test in 0.002s 2023-01-11T22:14:04.9109655Z 2023-01-11T22:14:04.9109761Z OK (skipped=1) 2023-01-11T22:14:04.9109780Z 2023-01-11T22:14:04.9109903Z Generating XML reports... 2023-01-11T22:14:04.9110346Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215525.xml 2023-01-11T22:14:04.9110752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9110933Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9111305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9111490Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9111509Z 2023-01-11T22:14:04.9111603Z Running tests... 2023-01-11T22:14:04.9111863Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9112171Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9112431Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2023-01-11T22:14:04.9112451Z 2023-01-11T22:14:04.9112707Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9112819Z Ran 1 test in 0.002s 2023-01-11T22:14:04.9112838Z 2023-01-11T22:14:04.9112944Z OK (skipped=1) 2023-01-11T22:14:04.9112963Z 2023-01-11T22:14:04.9113083Z Generating XML reports... 2023-01-11T22:14:04.9113522Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215528.xml 2023-01-11T22:14:04.9113865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9114041Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9114414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9114601Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9114620Z 2023-01-11T22:14:04.9114725Z Running tests... 2023-01-11T22:14:04.9114987Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9115295Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9115536Z test_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9115752Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37719 2023-01-11T22:14:04.9115947Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37720 2023-01-11T22:14:04.9116308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9116533Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9116911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9117099Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9117455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9117624Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9117991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9118161Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9118403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9118646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9119044Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9119432Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9119702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9119933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9120269Z STAGE:2023-01-11 21:55:34 37720:37720 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9120587Z STAGE:2023-01-11 21:55:34 37719:37719 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9120901Z STAGE:2023-01-11 21:55:34 37719:37719 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9121234Z STAGE:2023-01-11 21:55:34 37720:37720 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9121576Z STAGE:2023-01-11 21:55:34 37719:37719 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9121918Z STAGE:2023-01-11 21:55:34 37720:37720 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9122242Z STAGE:2023-01-11 21:55:34 37719:37719 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9122561Z STAGE:2023-01-11 21:55:34 37720:37720 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9122892Z STAGE:2023-01-11 21:55:34 37720:37720 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9123233Z STAGE:2023-01-11 21:55:34 37720:37720 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9123564Z STAGE:2023-01-11 21:55:34 37719:37719 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9123886Z STAGE:2023-01-11 21:55:34 37719:37719 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9124208Z STAGE:2023-01-11 21:55:34 37720:37720 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9124532Z STAGE:2023-01-11 21:55:34 37719:37719 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9124862Z STAGE:2023-01-11 21:55:34 37719:37719 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9125183Z STAGE:2023-01-11 21:55:34 37720:37720 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9125521Z STAGE:2023-01-11 21:55:34 37719:37719 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9125859Z STAGE:2023-01-11 21:55:34 37720:37720 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9126298Z STAGE:2023-01-11 21:55:34 37719:37719 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9126613Z STAGE:2023-01-11 21:55:34 37720:37720 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9126920Z STAGE:2023-01-11 21:55:34 37720:37720 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9127261Z STAGE:2023-01-11 21:55:34 37720:37720 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9127589Z STAGE:2023-01-11 21:55:34 37719:37719 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9127926Z STAGE:2023-01-11 21:55:34 37719:37719 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9128247Z STAGE:2023-01-11 21:55:34 37720:37720 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9128562Z STAGE:2023-01-11 21:55:34 37719:37719 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9128893Z STAGE:2023-01-11 21:55:34 37719:37719 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9129214Z STAGE:2023-01-11 21:55:34 37720:37720 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9129552Z STAGE:2023-01-11 21:55:34 37719:37719 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9129918Z STAGE:2023-01-11 21:55:34 37720:37720 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9130249Z STAGE:2023-01-11 21:55:34 37719:37719 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9130744Z STAGE:2023-01-11 21:55:34 37720:37720 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9131073Z STAGE:2023-01-11 21:55:34 37720:37720 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9131410Z STAGE:2023-01-11 21:55:34 37720:37720 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9131739Z STAGE:2023-01-11 21:55:34 37719:37719 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9132076Z STAGE:2023-01-11 21:55:34 37719:37719 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9132396Z STAGE:2023-01-11 21:55:34 37720:37720 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9132717Z STAGE:2023-01-11 21:55:34 37719:37719 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9133027Z STAGE:2023-01-11 21:55:34 37719:37719 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9133347Z STAGE:2023-01-11 21:55:34 37720:37720 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9133684Z STAGE:2023-01-11 21:55:34 37719:37719 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9134027Z STAGE:2023-01-11 21:55:34 37720:37720 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9134347Z STAGE:2023-01-11 21:55:34 37719:37719 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9134667Z STAGE:2023-01-11 21:55:34 37720:37720 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9134993Z STAGE:2023-01-11 21:55:34 37720:37720 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9135334Z STAGE:2023-01-11 21:55:34 37720:37720 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9135660Z STAGE:2023-01-11 21:55:34 37719:37719 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9135980Z STAGE:2023-01-11 21:55:34 37719:37719 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9136299Z STAGE:2023-01-11 21:55:34 37720:37720 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9136699Z STAGE:2023-01-11 21:55:34 37719:37719 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9137025Z STAGE:2023-01-11 21:55:34 37719:37719 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9137345Z STAGE:2023-01-11 21:55:34 37720:37720 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9137681Z STAGE:2023-01-11 21:55:34 37719:37719 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9138021Z STAGE:2023-01-11 21:55:34 37720:37720 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9138341Z STAGE:2023-01-11 21:55:34 37719:37719 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9138657Z STAGE:2023-01-11 21:55:34 37720:37720 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9138964Z STAGE:2023-01-11 21:55:34 37720:37720 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9139302Z STAGE:2023-01-11 21:55:34 37720:37720 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9139626Z STAGE:2023-01-11 21:55:34 37719:37719 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9139960Z STAGE:2023-01-11 21:55:34 37719:37719 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9140333Z STAGE:2023-01-11 21:55:34 37720:37720 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9140659Z STAGE:2023-01-11 21:55:34 37719:37719 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9140984Z STAGE:2023-01-11 21:55:34 37719:37719 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9141310Z STAGE:2023-01-11 21:55:34 37720:37720 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9141645Z STAGE:2023-01-11 21:55:34 37719:37719 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9141969Z STAGE:2023-01-11 21:55:34 37720:37720 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9142286Z STAGE:2023-01-11 21:55:34 37719:37719 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9142602Z STAGE:2023-01-11 21:55:34 37720:37720 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9142929Z STAGE:2023-01-11 21:55:34 37720:37720 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9143265Z STAGE:2023-01-11 21:55:34 37720:37720 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9143590Z STAGE:2023-01-11 21:55:34 37719:37719 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9143927Z STAGE:2023-01-11 21:55:34 37719:37719 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9144029Z ok (4.266s) 2023-01-11T22:14:04.9144052Z 2023-01-11T22:14:04.9144355Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9144452Z Ran 1 test in 4.267s 2023-01-11T22:14:04.9144471Z 2023-01-11T22:14:04.9144562Z OK 2023-01-11T22:14:04.9144581Z 2023-01-11T22:14:04.9144703Z Generating XML reports... 2023-01-11T22:14:04.9145152Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215530.xml 2023-01-11T22:14:04.9145523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9145695Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9146071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9146258Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9146278Z 2023-01-11T22:14:04.9146425Z Running tests... 2023-01-11T22:14:04.9146689Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9146996Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9147246Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9147994Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.653s) 2023-01-11T22:14:04.9148016Z 2023-01-11T22:14:04.9148273Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9148383Z Ran 1 test in 1.653s 2023-01-11T22:14:04.9148403Z 2023-01-11T22:14:04.9148509Z OK (skipped=1) 2023-01-11T22:14:04.9148527Z 2023-01-11T22:14:04.9148653Z Generating XML reports... 2023-01-11T22:14:04.9149097Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215537.xml 2023-01-11T22:14:04.9149446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9149619Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9150042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9150238Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9150258Z 2023-01-11T22:14:04.9150365Z Running tests... 2023-01-11T22:14:04.9150625Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9150933Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9151191Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9151394Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37866 2023-01-11T22:14:04.9151607Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37867 2023-01-11T22:14:04.9151970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9152146Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9152521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9152708Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9153118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9153287Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9153660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9153827Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9154068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9154308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9154706Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9155099Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9155325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9155547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9155842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T22:14:04.9156061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T22:14:04.9156460Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.9156854Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.9157185Z STAGE:2023-01-11 21:55:45 37866:37866 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9157508Z STAGE:2023-01-11 21:55:45 37867:37867 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9158047Z STAGE:2023-01-11 21:55:45 37867:37867 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:55:45 37866:37866 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9158071Z 2023-01-11T22:14:04.9158632Z STAGE:2023-01-11 21:55:45 37867:37867 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:55:45 37866:37866 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9158652Z 2023-01-11T22:14:04.9158969Z STAGE:2023-01-11 21:55:45 37866:37866 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9159328Z STAGE:2023-01-11 21:55:45 37867:37867 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9159667Z STAGE:2023-01-11 21:55:45 37867:37867 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9160008Z STAGE:2023-01-11 21:55:45 37867:37867 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9160319Z STAGE:2023-01-11 21:55:45 37866:37866 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9160664Z STAGE:2023-01-11 21:55:45 37866:37866 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9160984Z STAGE:2023-01-11 21:55:45 37867:37867 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9161299Z STAGE:2023-01-11 21:55:45 37866:37866 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9161623Z STAGE:2023-01-11 21:55:45 37866:37866 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9161946Z STAGE:2023-01-11 21:55:45 37867:37867 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9162287Z STAGE:2023-01-11 21:55:45 37866:37866 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9162623Z STAGE:2023-01-11 21:55:45 37867:37867 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9162945Z STAGE:2023-01-11 21:55:45 37866:37866 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9163251Z STAGE:2023-01-11 21:55:45 37867:37867 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9163577Z STAGE:2023-01-11 21:55:45 37867:37867 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9163913Z STAGE:2023-01-11 21:55:45 37867:37867 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9164240Z STAGE:2023-01-11 21:55:45 37866:37866 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9164580Z STAGE:2023-01-11 21:55:45 37866:37866 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9164902Z STAGE:2023-01-11 21:55:45 37867:37867 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9165217Z STAGE:2023-01-11 21:55:45 37866:37866 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9165546Z STAGE:2023-01-11 21:55:45 37866:37866 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9165940Z STAGE:2023-01-11 21:55:45 37867:37867 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9166261Z STAGE:2023-01-11 21:55:45 37866:37866 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9166598Z STAGE:2023-01-11 21:55:45 37867:37867 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9166921Z STAGE:2023-01-11 21:55:45 37866:37866 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9167242Z STAGE:2023-01-11 21:55:45 37867:37867 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9167565Z STAGE:2023-01-11 21:55:45 37867:37867 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9168112Z STAGE:2023-01-11 21:55:45 37867:37867 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:55:45 37866:37866 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9168135Z 2023-01-11T22:14:04.9168474Z STAGE:2023-01-11 21:55:45 37866:37866 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9168794Z STAGE:2023-01-11 21:55:45 37867:37867 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9169109Z STAGE:2023-01-11 21:55:45 37866:37866 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9169483Z STAGE:2023-01-11 21:55:45 37866:37866 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9169795Z STAGE:2023-01-11 21:55:45 37867:37867 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9170133Z STAGE:2023-01-11 21:55:45 37866:37866 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9170645Z STAGE:2023-01-11 21:55:45 37867:37867 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9170982Z STAGE:2023-01-11 21:55:45 37866:37866 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9171300Z STAGE:2023-01-11 21:55:45 37867:37867 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9171628Z STAGE:2023-01-11 21:55:45 37867:37867 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9172180Z STAGE:2023-01-11 21:55:45 37867:37867 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:55:45 37866:37866 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9172201Z 2023-01-11T22:14:04.9172542Z STAGE:2023-01-11 21:55:45 37866:37866 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9172861Z STAGE:2023-01-11 21:55:45 37867:37867 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9173174Z STAGE:2023-01-11 21:55:45 37866:37866 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9173487Z STAGE:2023-01-11 21:55:45 37866:37866 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9173809Z STAGE:2023-01-11 21:55:45 37867:37867 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9174149Z STAGE:2023-01-11 21:55:45 37866:37866 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9174486Z STAGE:2023-01-11 21:55:45 37867:37867 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9174805Z STAGE:2023-01-11 21:55:45 37866:37866 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9175121Z STAGE:2023-01-11 21:55:45 37867:37867 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9175448Z STAGE:2023-01-11 21:55:45 37867:37867 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9175997Z STAGE:2023-01-11 21:55:45 37867:37867 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:55:45 37866:37866 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9176091Z 2023-01-11T22:14:04.9176442Z STAGE:2023-01-11 21:55:45 37866:37866 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9176742Z STAGE:2023-01-11 21:55:45 37867:37867 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9177060Z STAGE:2023-01-11 21:55:45 37866:37866 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9177389Z STAGE:2023-01-11 21:55:45 37866:37866 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9177712Z STAGE:2023-01-11 21:55:45 37867:37867 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9178049Z STAGE:2023-01-11 21:55:45 37866:37866 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9178387Z STAGE:2023-01-11 21:55:45 37867:37867 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9178718Z STAGE:2023-01-11 21:55:45 37866:37866 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9179038Z STAGE:2023-01-11 21:55:45 37867:37867 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9179366Z STAGE:2023-01-11 21:55:45 37867:37867 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9179748Z STAGE:2023-01-11 21:55:45 37867:37867 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9180092Z STAGE:2023-01-11 21:55:45 37866:37866 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9180430Z STAGE:2023-01-11 21:55:45 37866:37866 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9180530Z ok (4.259s) 2023-01-11T22:14:04.9180551Z 2023-01-11T22:14:04.9180813Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9180930Z Ran 1 test in 4.259s 2023-01-11T22:14:04.9180949Z 2023-01-11T22:14:04.9181040Z OK 2023-01-11T22:14:04.9181060Z 2023-01-11T22:14:04.9181182Z Generating XML reports... 2023-01-11T22:14:04.9181628Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215541.xml 2023-01-11T22:14:04.9181979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9182156Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9182535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9182722Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9182741Z 2023-01-11T22:14:04.9182846Z Running tests... 2023-01-11T22:14:04.9183106Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9183418Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9183670Z test_broadcast_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9183868Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37985 2023-01-11T22:14:04.9184083Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37986 2023-01-11T22:14:04.9184451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9184626Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9185002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9185188Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9185547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9185772Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9186142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9186312Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9186556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9186795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9187190Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9187579Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9187806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9188034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9188192Z skip: Skipped due to small world size. (4.243s) 2023-01-11T22:14:04.9188211Z 2023-01-11T22:14:04.9188473Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9188568Z Ran 1 test in 4.244s 2023-01-11T22:14:04.9188587Z 2023-01-11T22:14:04.9188694Z OK (skipped=1) 2023-01-11T22:14:04.9188760Z 2023-01-11T22:14:04.9188891Z Generating XML reports... 2023-01-11T22:14:04.9189345Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215548.xml 2023-01-11T22:14:04.9189712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9189884Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9190263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9190451Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9190471Z 2023-01-11T22:14:04.9190578Z Running tests... 2023-01-11T22:14:04.9190823Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9191135Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9191392Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9191607Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38094 2023-01-11T22:14:04.9191820Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38095 2023-01-11T22:14:04.9192185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9192359Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9192734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9192906Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9193262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9193437Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9193808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9193992Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9194232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9194471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9194923Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9195313Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9195523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9195748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9196511Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1478: UserWarning: torch.distributed.broadcast_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2023-01-11T22:14:04.9196624Z warnings.warn( 2023-01-11T22:14:04.9197377Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1478: UserWarning: torch.distributed.broadcast_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2023-01-11T22:14:04.9197490Z warnings.warn( 2023-01-11T22:14:04.9197589Z ok (5.770s) 2023-01-11T22:14:04.9197609Z 2023-01-11T22:14:04.9197873Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9198028Z Ran 1 test in 5.770s 2023-01-11T22:14:04.9198048Z 2023-01-11T22:14:04.9198126Z OK 2023-01-11T22:14:04.9198144Z 2023-01-11T22:14:04.9198271Z Generating XML reports... 2023-01-11T22:14:04.9198721Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215555.xml 2023-01-11T22:14:04.9199084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9199262Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9199636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9199822Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9199841Z 2023-01-11T22:14:04.9199946Z Running tests... 2023-01-11T22:14:04.9200206Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9200504Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9200763Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9201496Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.604s) 2023-01-11T22:14:04.9201519Z 2023-01-11T22:14:04.9201780Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9201888Z Ran 1 test in 1.605s 2023-01-11T22:14:04.9201908Z 2023-01-11T22:14:04.9202012Z OK (skipped=1) 2023-01-11T22:14:04.9202031Z 2023-01-11T22:14:04.9202152Z Generating XML reports... 2023-01-11T22:14:04.9202597Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215603.xml 2023-01-11T22:14:04.9202964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9203120Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9203495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9203683Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9203752Z 2023-01-11T22:14:04.9203863Z Running tests... 2023-01-11T22:14:04.9204125Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9204433Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9204742Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9205482Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85012 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.659s) 2023-01-11T22:14:04.9205503Z 2023-01-11T22:14:04.9205760Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9205871Z Ran 1 test in 1.659s 2023-01-11T22:14:04.9205893Z 2023-01-11T22:14:04.9205982Z OK (skipped=1) 2023-01-11T22:14:04.9206001Z 2023-01-11T22:14:04.9206122Z Generating XML reports... 2023-01-11T22:14:04.9206566Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215607.xml 2023-01-11T22:14:04.9206932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9207153Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9207541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9207728Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9207748Z 2023-01-11T22:14:04.9207854Z Running tests... 2023-01-11T22:14:04.9208114Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9208404Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9208720Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9209463Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85339 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.611s) 2023-01-11T22:14:04.9209483Z 2023-01-11T22:14:04.9209739Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9209849Z Ran 1 test in 1.611s 2023-01-11T22:14:04.9209868Z 2023-01-11T22:14:04.9209972Z OK (skipped=1) 2023-01-11T22:14:04.9209991Z 2023-01-11T22:14:04.9210110Z Generating XML reports... 2023-01-11T22:14:04.9210728Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215611.xml 2023-01-11T22:14:04.9211109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9211283Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9211642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9211833Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9211853Z 2023-01-11T22:14:04.9211958Z Running tests... 2023-01-11T22:14:04.9212218Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9212525Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9212795Z test_ddp_apply_optim_in_backward (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9213011Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38307 2023-01-11T22:14:04.9213316Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38308 2023-01-11T22:14:04.9213668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9213841Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9214219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9214405Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9214766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9214934Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9215299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9215488Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9215730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9215953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9216345Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9216793Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9217029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9217251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9218024Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired. 2023-01-11T22:14:04.9218140Z warnings.warn( 2023-01-11T22:14:04.9218908Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired. 2023-01-11T22:14:04.9219017Z warnings.warn( 2023-01-11T22:14:04.9219248Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9219455Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9219687Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9219919Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9220148Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9220369Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9220587Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9220812Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9220912Z ok (7.628s) 2023-01-11T22:14:04.9220931Z 2023-01-11T22:14:04.9221181Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9221294Z Ran 1 test in 7.628s 2023-01-11T22:14:04.9221314Z 2023-01-11T22:14:04.9221406Z OK 2023-01-11T22:14:04.9221425Z 2023-01-11T22:14:04.9221548Z Generating XML reports... 2023-01-11T22:14:04.9221994Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215615.xml 2023-01-11T22:14:04.9222422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9222596Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9222972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9223163Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9223182Z 2023-01-11T22:14:04.9223273Z Running tests... 2023-01-11T22:14:04.9223533Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9223844Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9224145Z test_ddp_apply_optim_in_backward_grad_as_bucket_view_false (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9224366Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38422 2023-01-11T22:14:04.9224580Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38423 2023-01-11T22:14:04.9224940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9225111Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9225516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9225707Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9226069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9226240Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9226610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9226797Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9227036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9227275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9227672Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9228046Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9228271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9228495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9229271Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired. 2023-01-11T22:14:04.9229383Z warnings.warn( 2023-01-11T22:14:04.9230146Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired. 2023-01-11T22:14:04.9230254Z warnings.warn( 2023-01-11T22:14:04.9230485Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9230708Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9231006Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9231217Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9231318Z ok (6.807s) 2023-01-11T22:14:04.9231337Z 2023-01-11T22:14:04.9231604Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9231713Z Ran 1 test in 6.808s 2023-01-11T22:14:04.9231733Z 2023-01-11T22:14:04.9231825Z OK 2023-01-11T22:14:04.9231847Z 2023-01-11T22:14:04.9231969Z Generating XML reports... 2023-01-11T22:14:04.9232414Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215626.xml 2023-01-11T22:14:04.9232779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9232936Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9233313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9233498Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9233517Z 2023-01-11T22:14:04.9233623Z Running tests... 2023-01-11T22:14:04.9233887Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9234195Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9234532Z test_ddp_apply_optim_in_backward_ignored_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9234754Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38537 2023-01-11T22:14:04.9234967Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38538 2023-01-11T22:14:04.9235320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9235496Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9235868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9236054Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9236418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9236591Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9236956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9237140Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9237362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9237601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9238001Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9238390Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9238618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9238844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9239613Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired. 2023-01-11T22:14:04.9239725Z warnings.warn( 2023-01-11T22:14:04.9240543Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired. 2023-01-11T22:14:04.9240651Z warnings.warn( 2023-01-11T22:14:04.9240734Z ok (7.310s) 2023-01-11T22:14:04.9240771Z 2023-01-11T22:14:04.9241019Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9241130Z Ran 1 test in 7.310s 2023-01-11T22:14:04.9241149Z 2023-01-11T22:14:04.9241241Z OK 2023-01-11T22:14:04.9241260Z 2023-01-11T22:14:04.9241381Z Generating XML reports... 2023-01-11T22:14:04.9241827Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215635.xml 2023-01-11T22:14:04.9242191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9242369Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9242745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9242917Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9242936Z 2023-01-11T22:14:04.9243043Z Running tests... 2023-01-11T22:14:04.9243350Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9243670Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9243930Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9244145Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38654 2023-01-11T22:14:04.9244394Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38655 2023-01-11T22:14:04.9244769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9244922Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9245296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9245481Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9245841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9246012Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9246382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9246564Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9246807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9247045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9247425Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9247816Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9248042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9248263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9248496Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9248730Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9248888Z ok (6.252s) 2023-01-11T22:14:04.9248907Z 2023-01-11T22:14:04.9249172Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9249266Z Ran 1 test in 6.252s 2023-01-11T22:14:04.9249303Z 2023-01-11T22:14:04.9249377Z OK 2023-01-11T22:14:04.9249396Z 2023-01-11T22:14:04.9249519Z Generating XML reports... 2023-01-11T22:14:04.9249966Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215645.xml 2023-01-11T22:14:04.9250498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9250685Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9251067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9251253Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9251276Z 2023-01-11T22:14:04.9251383Z Running tests... 2023-01-11T22:14:04.9251627Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9251937Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9252207Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9252422Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38769 2023-01-11T22:14:04.9252717Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38770 2023-01-11T22:14:04.9253098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9253267Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9253642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9253814Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9254175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9254346Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9254714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9254900Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9255142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9255383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9255776Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9256172Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9256379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9256602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9256834Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9257065Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9257287Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9257517Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9257618Z ok (6.259s) 2023-01-11T22:14:04.9257638Z 2023-01-11T22:14:04.9257902Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9258087Z Ran 1 test in 6.260s 2023-01-11T22:14:04.9258107Z 2023-01-11T22:14:04.9258181Z OK 2023-01-11T22:14:04.9258200Z 2023-01-11T22:14:04.9258323Z Generating XML reports... 2023-01-11T22:14:04.9258775Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215654.xml 2023-01-11T22:14:04.9259141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9259318Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9259695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9259881Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9259900Z 2023-01-11T22:14:04.9260006Z Running tests... 2023-01-11T22:14:04.9260248Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9260558Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9260824Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9261612Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78641 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.657s) 2023-01-11T22:14:04.9261634Z 2023-01-11T22:14:04.9261900Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9262010Z Ran 1 test in 1.657s 2023-01-11T22:14:04.9262030Z 2023-01-11T22:14:04.9262136Z OK (skipped=1) 2023-01-11T22:14:04.9262154Z 2023-01-11T22:14:04.9262277Z Generating XML reports... 2023-01-11T22:14:04.9262721Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215703.xml 2023-01-11T22:14:04.9263091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9263247Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9263620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9263810Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9263829Z 2023-01-11T22:14:04.9263934Z Running tests... 2023-01-11T22:14:04.9264195Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9264502Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9264786Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9265519Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.598s) 2023-01-11T22:14:04.9265543Z 2023-01-11T22:14:04.9265801Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9265896Z Ran 1 test in 1.599s 2023-01-11T22:14:04.9265931Z 2023-01-11T22:14:04.9266025Z OK (skipped=1) 2023-01-11T22:14:04.9266044Z 2023-01-11T22:14:04.9266167Z Generating XML reports... 2023-01-11T22:14:04.9266608Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215707.xml 2023-01-11T22:14:04.9266971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9267142Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9267581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9267765Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9267785Z 2023-01-11T22:14:04.9267892Z Running tests... 2023-01-11T22:14:04.9268134Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9268444Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9268722Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9268937Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38952 2023-01-11T22:14:04.9269148Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38953 2023-01-11T22:14:04.9269514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9269689Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9270066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9270253Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9270694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9270872Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9271239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9271423Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9271664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9271911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9272306Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9272695Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9272906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9273131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9273339Z 2023-01-11T22:14:04.9273438Z ok (5.714s) 2023-01-11T22:14:04.9273458Z 2023-01-11T22:14:04.9273721Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9273830Z Ran 1 test in 5.714s 2023-01-11T22:14:04.9273849Z 2023-01-11T22:14:04.9273941Z OK 2023-01-11T22:14:04.9273963Z 2023-01-11T22:14:04.9274085Z Generating XML reports... 2023-01-11T22:14:04.9274529Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215711.xml 2023-01-11T22:14:04.9274882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9275055Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9275435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9275623Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9275642Z 2023-01-11T22:14:04.9275749Z Running tests... 2023-01-11T22:14:04.9276008Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9276319Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9276675Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9276874Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39063 2023-01-11T22:14:04.9277088Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39064 2023-01-11T22:14:04.9277459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9277637Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9278007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9278196Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9278557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9278734Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9279104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9279274Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9279513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9279800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9280208Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9280599Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9280823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9281049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9281150Z ok (5.721s) 2023-01-11T22:14:04.9281169Z 2023-01-11T22:14:04.9281433Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9281528Z Ran 1 test in 5.722s 2023-01-11T22:14:04.9281546Z 2023-01-11T22:14:04.9281639Z OK 2023-01-11T22:14:04.9281659Z 2023-01-11T22:14:04.9281778Z Generating XML reports... 2023-01-11T22:14:04.9282225Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215719.xml 2023-01-11T22:14:04.9282593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9282765Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9283139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9283328Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9283347Z 2023-01-11T22:14:04.9283438Z Running tests... 2023-01-11T22:14:04.9283702Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9284012Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9295535Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9295837Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39174 2023-01-11T22:14:04.9296070Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39175 2023-01-11T22:14:04.9296507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9296689Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9297077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9297432Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9297832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9298011Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9298412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9298608Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9298860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9299109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9299523Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9299922Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9300158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9300392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9300706Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9300960Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9301062Z ok (6.255s) 2023-01-11T22:14:04.9301085Z 2023-01-11T22:14:04.9301371Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9301483Z Ran 1 test in 6.256s 2023-01-11T22:14:04.9301503Z 2023-01-11T22:14:04.9301596Z OK 2023-01-11T22:14:04.9301615Z 2023-01-11T22:14:04.9301729Z Generating XML reports... 2023-01-11T22:14:04.9302202Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215727.xml 2023-01-11T22:14:04.9302590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9302770Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9303172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9303368Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9303388Z 2023-01-11T22:14:04.9303497Z Running tests... 2023-01-11T22:14:04.9303773Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9304084Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9304381Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9304616Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39289 2023-01-11T22:14:04.9304840Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39290 2023-01-11T22:14:04.9305224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9305407Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9305801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9305996Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9306378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9306540Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9307023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9307218Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9307469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9307723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9308142Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9308552Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9308789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9309024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9309893Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-01-11T22:14:04.9310007Z ok (6.257s) 2023-01-11T22:14:04.9310045Z 2023-01-11T22:14:04.9310308Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9310423Z Ran 1 test in 6.257s 2023-01-11T22:14:04.9310443Z 2023-01-11T22:14:04.9310535Z OK 2023-01-11T22:14:04.9310554Z 2023-01-11T22:14:04.9310679Z Generating XML reports... 2023-01-11T22:14:04.9311157Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215736.xml 2023-01-11T22:14:04.9311543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9311724Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9312120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9312302Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9312321Z 2023-01-11T22:14:04.9312431Z Running tests... 2023-01-11T22:14:04.9312705Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9313033Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9313323Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9314118Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78235 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.635s) 2023-01-11T22:14:04.9314138Z 2023-01-11T22:14:04.9314415Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9314533Z Ran 1 test in 1.635s 2023-01-11T22:14:04.9314553Z 2023-01-11T22:14:04.9314662Z OK (skipped=1) 2023-01-11T22:14:04.9314681Z 2023-01-11T22:14:04.9314791Z Generating XML reports... 2023-01-11T22:14:04.9315261Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215745.xml 2023-01-11T22:14:04.9315647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9315886Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9316287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9316480Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9316500Z 2023-01-11T22:14:04.9316608Z Running tests... 2023-01-11T22:14:04.9316891Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9317224Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9317473Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9317696Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39438 2023-01-11T22:14:04.9317921Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39439 2023-01-11T22:14:04.9318306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9318488Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9318885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9319080Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9319511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9319680Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9320079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9320273Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9320524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9320779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9321196Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9321608Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9321849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9322083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9323038Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2023-01-11T22:14:04.9323969Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2023-01-11T22:14:04.9325195Z /opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py:197: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1134.) 2023-01-11T22:14:04.9325487Z Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2023-01-11T22:14:04.9326715Z /opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py:197: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1134.) 2023-01-11T22:14:04.9326952Z Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2023-01-11T22:14:04.9327192Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9327418Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9328358Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2023-01-11T22:14:04.9329335Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2023-01-11T22:14:04.9330484Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2023-01-11T22:14:04.9331464Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2023-01-11T22:14:04.9332400Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2023-01-11T22:14:04.9333323Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2023-01-11T22:14:04.9334261Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2023-01-11T22:14:04.9335208Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2023-01-11T22:14:04.9336236Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2023-01-11T22:14:04.9337170Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2023-01-11T22:14:04.9337259Z ok (4.112s) 2023-01-11T22:14:04.9337297Z 2023-01-11T22:14:04.9337557Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9337674Z Ran 1 test in 4.112s 2023-01-11T22:14:04.9337695Z 2023-01-11T22:14:04.9337786Z OK 2023-01-11T22:14:04.9337805Z 2023-01-11T22:14:04.9337930Z Generating XML reports... 2023-01-11T22:14:04.9338405Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215749.xml 2023-01-11T22:14:04.9338854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9339042Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9339447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9339627Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9339647Z 2023-01-11T22:14:04.9339757Z Running tests... 2023-01-11T22:14:04.9340033Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9340366Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9340618Z test_ddp_device (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9341404Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77324 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.678s) 2023-01-11T22:14:04.9341426Z 2023-01-11T22:14:04.9341701Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9341816Z Ran 1 test in 1.678s 2023-01-11T22:14:04.9341835Z 2023-01-11T22:14:04.9341944Z OK (skipped=1) 2023-01-11T22:14:04.9341963Z 2023-01-11T22:14:04.9342071Z Generating XML reports... 2023-01-11T22:14:04.9342544Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215756.xml 2023-01-11T22:14:04.9342938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9343119Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9343519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9343718Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9343737Z 2023-01-11T22:14:04.9343848Z Running tests... 2023-01-11T22:14:04.9344126Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9344504Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9344758Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9345044Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39585 2023-01-11T22:14:04.9345259Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39586 2023-01-11T22:14:04.9345636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9345805Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9346184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9346373Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9346734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9346886Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9347254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9347445Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9347690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9347929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9348369Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9348769Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9348994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9349218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9349998Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1331: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior. 2023-01-11T22:14:04.9350311Z warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes " 2023-01-11T22:14:04.9351092Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1331: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior. 2023-01-11T22:14:04.9351417Z warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes " 2023-01-11T22:14:04.9351519Z ok (6.268s) 2023-01-11T22:14:04.9351539Z 2023-01-11T22:14:04.9351809Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9351921Z Ran 1 test in 6.269s 2023-01-11T22:14:04.9351941Z 2023-01-11T22:14:04.9352032Z OK 2023-01-11T22:14:04.9352051Z 2023-01-11T22:14:04.9352173Z Generating XML reports... 2023-01-11T22:14:04.9352622Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215800.xml 2023-01-11T22:14:04.9352989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9353144Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9353519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9353706Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9353726Z 2023-01-11T22:14:04.9353832Z Running tests... 2023-01-11T22:14:04.9354150Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9354460Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9354727Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9355466Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78685 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.628s) 2023-01-11T22:14:04.9355486Z 2023-01-11T22:14:04.9355746Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9355846Z Ran 1 test in 1.628s 2023-01-11T22:14:04.9355882Z 2023-01-11T22:14:04.9355972Z OK (skipped=1) 2023-01-11T22:14:04.9355991Z 2023-01-11T22:14:04.9356114Z Generating XML reports... 2023-01-11T22:14:04.9356563Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215809.xml 2023-01-11T22:14:04.9356928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9357101Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9357520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9357714Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9357734Z 2023-01-11T22:14:04.9357842Z Running tests... 2023-01-11T22:14:04.9358086Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9358393Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9358659Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9359396Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.645s) 2023-01-11T22:14:04.9359416Z 2023-01-11T22:14:04.9359668Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9359781Z Ran 1 test in 1.645s 2023-01-11T22:14:04.9359800Z 2023-01-11T22:14:04.9359905Z OK (skipped=1) 2023-01-11T22:14:04.9359924Z 2023-01-11T22:14:04.9360043Z Generating XML reports... 2023-01-11T22:14:04.9360487Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215813.xml 2023-01-11T22:14:04.9360853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9361012Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9361385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9361570Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9361590Z 2023-01-11T22:14:04.9361693Z Running tests... 2023-01-11T22:14:04.9361949Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9362258Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9362542Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9362758Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39798 2023-01-11T22:14:04.9362955Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39799 2023-01-11T22:14:04.9363381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9363552Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9363925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9364112Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9364471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9364639Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9365007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9365193Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9365418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9365661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9366058Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9366450Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9366722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9366955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9367189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T22:14:04.9367424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T22:14:04.9367827Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.9368196Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.9368429Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9368662Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9368893Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9369121Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9369222Z ok (6.502s) 2023-01-11T22:14:04.9369242Z 2023-01-11T22:14:04.9369505Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9369617Z Ran 1 test in 6.502s 2023-01-11T22:14:04.9369636Z 2023-01-11T22:14:04.9369715Z OK 2023-01-11T22:14:04.9369751Z 2023-01-11T22:14:04.9369855Z Generating XML reports... 2023-01-11T22:14:04.9370485Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215817.xml 2023-01-11T22:14:04.9370871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9371046Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9371425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9371614Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9371634Z 2023-01-11T22:14:04.9371742Z Running tests... 2023-01-11T22:14:04.9372003Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9372295Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9372667Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9372884Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39919 2023-01-11T22:14:04.9373100Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39920 2023-01-11T22:14:04.9373469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9373646Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9374022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9374210Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9374548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9374722Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9375093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9375277Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9375521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9375821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9376233Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9376623Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9376847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9377057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9377332Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2023-01-11T22:14:04.9377601Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2023-01-11T22:14:04.9377833Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9378067Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9378297Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9378522Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9378793Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2023-01-11T22:14:04.9379065Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2023-01-11T22:14:04.9379314Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2023-01-11T22:14:04.9379585Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2023-01-11T22:14:04.9379811Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9380043Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9380272Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9380499Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9380767Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2023-01-11T22:14:04.9381093Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2023-01-11T22:14:04.9381365Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations 2023-01-11T22:14:04.9381610Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations 2023-01-11T22:14:04.9381839Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9382068Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9382291Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9382516Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9382617Z ok (7.063s) 2023-01-11T22:14:04.9382637Z 2023-01-11T22:14:04.9382907Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9383023Z Ran 1 test in 7.063s 2023-01-11T22:14:04.9383042Z 2023-01-11T22:14:04.9383117Z OK 2023-01-11T22:14:04.9383153Z 2023-01-11T22:14:04.9383258Z Generating XML reports... 2023-01-11T22:14:04.9383704Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215826.xml 2023-01-11T22:14:04.9384119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9384299Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9384676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9384864Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9384884Z 2023-01-11T22:14:04.9384990Z Running tests... 2023-01-11T22:14:04.9385249Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9385546Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9385812Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9386558Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77378 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.645s) 2023-01-11T22:14:04.9386579Z 2023-01-11T22:14:04.9386837Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9386947Z Ran 1 test in 1.645s 2023-01-11T22:14:04.9386967Z 2023-01-11T22:14:04.9387073Z OK (skipped=1) 2023-01-11T22:14:04.9387092Z 2023-01-11T22:14:04.9387214Z Generating XML reports... 2023-01-11T22:14:04.9387655Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215836.xml 2023-01-11T22:14:04.9388027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9388200Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9388561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9388752Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9388771Z 2023-01-11T22:14:04.9388879Z Running tests... 2023-01-11T22:14:04.9389137Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9389446Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9389711Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9389993Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40068 2023-01-11T22:14:04.9390205Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40069 2023-01-11T22:14:04.9390557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9390727Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9391097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9391267Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9391639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9391825Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9392194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9392383Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9392627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9392851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9393291Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9393696Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9393919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9394143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9394681Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2023-01-11T22:14:04.9395220Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2023-01-11T22:14:04.9395452Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9395674Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9395944Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations. 2023-01-11T22:14:04.9396217Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations. 2023-01-11T22:14:04.9396493Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created. 2023-01-11T22:14:04.9396782Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created. 2023-01-11T22:14:04.9397105Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0 2023-01-11T22:14:04.9397425Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0 2023-01-11T22:14:04.9397742Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively. 2023-01-11T22:14:04.9398113Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively. 2023-01-11T22:14:04.9398347Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9398576Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9398678Z ok (6.211s) 2023-01-11T22:14:04.9398698Z 2023-01-11T22:14:04.9398955Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9399068Z Ran 1 test in 6.212s 2023-01-11T22:14:04.9399087Z 2023-01-11T22:14:04.9399177Z OK 2023-01-11T22:14:04.9399196Z 2023-01-11T22:14:04.9399317Z Generating XML reports... 2023-01-11T22:14:04.9399763Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215840.xml 2023-01-11T22:14:04.9400127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9400303Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9400676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9400864Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9400884Z 2023-01-11T22:14:04.9400973Z Running tests... 2023-01-11T22:14:04.9401281Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9401599Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9401906Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9402121Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40183 2023-01-11T22:14:04.9402336Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40184 2023-01-11T22:14:04.9402701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9402870Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9403225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9403415Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9403773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9403942Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9404311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9404493Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9404740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9404979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9405375Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9405749Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9405974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9406197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9406429Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9406663Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9406946Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9407174Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9407274Z ok (6.414s) 2023-01-11T22:14:04.9407293Z 2023-01-11T22:14:04.9407559Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9407654Z Ran 1 test in 6.414s 2023-01-11T22:14:04.9407676Z 2023-01-11T22:14:04.9407767Z OK 2023-01-11T22:14:04.9407786Z 2023-01-11T22:14:04.9407906Z Generating XML reports... 2023-01-11T22:14:04.9408357Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215849.xml 2023-01-11T22:14:04.9408723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9408896Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9409278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9409466Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9409486Z 2023-01-11T22:14:04.9409575Z Running tests... 2023-01-11T22:14:04.9409837Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9410196Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9410686Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9410909Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40328 2023-01-11T22:14:04.9411123Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40329 2023-01-11T22:14:04.9411496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9411674Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9412051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9412223Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9412583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9412752Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9413120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9413301Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9413543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9413787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9414181Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9414570Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9414783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9415005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9415235Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9415464Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9415692Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9416009Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9416110Z ok (6.448s) 2023-01-11T22:14:04.9416131Z 2023-01-11T22:14:04.9416399Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9416493Z Ran 1 test in 6.448s 2023-01-11T22:14:04.9416513Z 2023-01-11T22:14:04.9416604Z OK 2023-01-11T22:14:04.9416623Z 2023-01-11T22:14:04.9416745Z Generating XML reports... 2023-01-11T22:14:04.9417199Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215858.xml 2023-01-11T22:14:04.9417569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9417743Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9418117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9418310Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9418330Z 2023-01-11T22:14:04.9418437Z Running tests... 2023-01-11T22:14:04.9418684Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9418990Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9419422Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9419648Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40473 2023-01-11T22:14:04.9419860Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40474 2023-01-11T22:14:04.9420229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9420406Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9420778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9420947Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9421305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9421476Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9421841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9422028Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9422267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9422506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9422906Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9423299Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9423508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9423736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9423970Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9424199Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9424423Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9424646Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9424853Z ok (6.439s) 2023-01-11T22:14:04.9424874Z 2023-01-11T22:14:04.9425142Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9425235Z Ran 1 test in 6.440s 2023-01-11T22:14:04.9425272Z 2023-01-11T22:14:04.9425346Z OK 2023-01-11T22:14:04.9425364Z 2023-01-11T22:14:04.9425485Z Generating XML reports... 2023-01-11T22:14:04.9425936Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215907.xml 2023-01-11T22:14:04.9426303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9426476Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9426849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9427035Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9427058Z 2023-01-11T22:14:04.9427165Z Running tests... 2023-01-11T22:14:04.9427408Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9427718Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9428128Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9428350Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40618 2023-01-11T22:14:04.9428563Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40619 2023-01-11T22:14:04.9428930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9429102Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9429476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9429668Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9430013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9430187Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9430558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9430742Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9430984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9431224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9431624Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9432021Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9432246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9432455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9432690Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9432920Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9433144Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9433370Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9433470Z ok (6.451s) 2023-01-11T22:14:04.9433490Z 2023-01-11T22:14:04.9433827Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9433938Z Ran 1 test in 6.451s 2023-01-11T22:14:04.9433958Z 2023-01-11T22:14:04.9434031Z OK 2023-01-11T22:14:04.9434050Z 2023-01-11T22:14:04.9434173Z Generating XML reports... 2023-01-11T22:14:04.9434623Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215916.xml 2023-01-11T22:14:04.9434995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9435167Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9435541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9438174Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9438194Z 2023-01-11T22:14:04.9438303Z Running tests... 2023-01-11T22:14:04.9438580Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9438875Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9439239Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9439513Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40763 2023-01-11T22:14:04.9439733Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40764 2023-01-11T22:14:04.9440103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9440274Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9440651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9440842Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9441208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9441362Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9441728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9441916Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9442159Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9442398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9442793Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9443185Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9443407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9443631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9443846Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9444075Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9444302Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9444575Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9444679Z ok (6.450s) 2023-01-11T22:14:04.9444699Z 2023-01-11T22:14:04.9444966Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9445134Z Ran 1 test in 6.450s 2023-01-11T22:14:04.9445153Z 2023-01-11T22:14:04.9445244Z OK 2023-01-11T22:14:04.9445263Z 2023-01-11T22:14:04.9445368Z Generating XML reports... 2023-01-11T22:14:04.9445819Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215925.xml 2023-01-11T22:14:04.9446187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9446365Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9446740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9446928Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9446947Z 2023-01-11T22:14:04.9447053Z Running tests... 2023-01-11T22:14:04.9447313Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9447609Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9447970Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9448187Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40908 2023-01-11T22:14:04.9448449Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40909 2023-01-11T22:14:04.9448824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9448995Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9449371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9449560Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9449925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9450077Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9450614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9450805Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9451057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9451299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9451702Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9452091Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9452319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9452544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9452758Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9453028Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9453262Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9453490Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9453592Z ok (6.410s) 2023-01-11T22:14:04.9453613Z 2023-01-11T22:14:04.9453879Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9453989Z Ran 1 test in 6.410s 2023-01-11T22:14:04.9454008Z 2023-01-11T22:14:04.9454099Z OK 2023-01-11T22:14:04.9454199Z 2023-01-11T22:14:04.9454312Z Generating XML reports... 2023-01-11T22:14:04.9454767Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215934.xml 2023-01-11T22:14:04.9455133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9455304Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9455682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9455870Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9455890Z 2023-01-11T22:14:04.9455995Z Running tests... 2023-01-11T22:14:04.9456255Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9456564Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9456912Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9457128Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41053 2023-01-11T22:14:04.9457341Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41054 2023-01-11T22:14:04.9457772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9457956Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9458334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9458525Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9458883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9459057Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9459409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9459595Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9459836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9460078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9460473Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9460863Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9461088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9461313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9461529Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9461759Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9461987Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9462211Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9462311Z ok (6.537s) 2023-01-11T22:14:04.9462331Z 2023-01-11T22:14:04.9462599Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9462710Z Ran 1 test in 6.537s 2023-01-11T22:14:04.9462729Z 2023-01-11T22:14:04.9462820Z OK 2023-01-11T22:14:04.9462839Z 2023-01-11T22:14:04.9462960Z Generating XML reports... 2023-01-11T22:14:04.9463448Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215943.xml 2023-01-11T22:14:04.9463817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9463990Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9464365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9464550Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9464569Z 2023-01-11T22:14:04.9464676Z Running tests... 2023-01-11T22:14:04.9464933Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9465241Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9465585Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9465806Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41198 2023-01-11T22:14:04.9466018Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41199 2023-01-11T22:14:04.9466382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9466598Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9466985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9467174Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9467532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9467702Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9468059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9468242Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9468483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9468724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9469118Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9469508Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9469732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9469955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9470190Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9470404Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9470626Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9470857Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9470961Z ok (6.410s) 2023-01-11T22:14:04.9470981Z 2023-01-11T22:14:04.9471244Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9471353Z Ran 1 test in 6.410s 2023-01-11T22:14:04.9471372Z 2023-01-11T22:14:04.9471463Z OK 2023-01-11T22:14:04.9471482Z 2023-01-11T22:14:04.9471603Z Generating XML reports... 2023-01-11T22:14:04.9472034Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215952.xml 2023-01-11T22:14:04.9472458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9472630Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9473008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9473196Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9473219Z 2023-01-11T22:14:04.9473324Z Running tests... 2023-01-11T22:14:04.9473582Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9473890Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9474252Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9474454Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41343 2023-01-11T22:14:04.9474665Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41344 2023-01-11T22:14:04.9475029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9475200Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9475623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9475816Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9476176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9476347Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9476718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9476888Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9477129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9477366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9477764Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9478155Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9478380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9478600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9478830Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9479045Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9479270Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9479497Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9479596Z ok (6.456s) 2023-01-11T22:14:04.9479618Z 2023-01-11T22:14:04.9479884Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9479994Z Ran 1 test in 6.456s 2023-01-11T22:14:04.9480013Z 2023-01-11T22:14:04.9480104Z OK 2023-01-11T22:14:04.9480123Z 2023-01-11T22:14:04.9480246Z Generating XML reports... 2023-01-11T22:14:04.9480690Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220001.xml 2023-01-11T22:14:04.9481039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9481267Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9481642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9481829Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9481849Z 2023-01-11T22:14:04.9481954Z Running tests... 2023-01-11T22:14:04.9482215Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9482524Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9482882Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9483097Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41488 2023-01-11T22:14:04.9483299Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41489 2023-01-11T22:14:04.9483662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9483833Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9484204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9484435Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9484803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9484972Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9485340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9485511Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9485753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9485992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9486388Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9486779Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9487005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9487226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9487460Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9487692Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9487899Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9488126Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9488226Z ok (6.414s) 2023-01-11T22:14:04.9488245Z 2023-01-11T22:14:04.9488508Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9488621Z Ran 1 test in 6.414s 2023-01-11T22:14:04.9488640Z 2023-01-11T22:14:04.9488731Z OK 2023-01-11T22:14:04.9488750Z 2023-01-11T22:14:04.9488871Z Generating XML reports... 2023-01-11T22:14:04.9489320Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220010.xml 2023-01-11T22:14:04.9489667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9489895Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9490275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9490675Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9490695Z 2023-01-11T22:14:04.9490802Z Running tests... 2023-01-11T22:14:04.9491064Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9491376Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9491686Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9491902Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41633 2023-01-11T22:14:04.9492094Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41634 2023-01-11T22:14:04.9492467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9492638Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9493010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9493194Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9493631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9493813Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9494184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9494351Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9494594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9494996Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9495236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9495622Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9495849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9496072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9496302Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9496531Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9496738Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9496969Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9497069Z ok (6.510s) 2023-01-11T22:14:04.9497088Z 2023-01-11T22:14:04.9497352Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9497461Z Ran 1 test in 6.510s 2023-01-11T22:14:04.9497480Z 2023-01-11T22:14:04.9497570Z OK 2023-01-11T22:14:04.9497589Z 2023-01-11T22:14:04.9497712Z Generating XML reports... 2023-01-11T22:14:04.9498161Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220019.xml 2023-01-11T22:14:04.9498525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9498682Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9499056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9499328Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9499347Z 2023-01-11T22:14:04.9499453Z Running tests... 2023-01-11T22:14:04.9499716Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9500025Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9500332Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9500551Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41778 2023-01-11T22:14:04.9500746Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41779 2023-01-11T22:14:04.9501112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9501282Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9501658Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9501843Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9502200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9502418Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9502795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9502978Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9503200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9503441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9503839Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9504230Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9504455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9504681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9504912Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9505140Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9505361Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9505572Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9505677Z ok (6.454s) 2023-01-11T22:14:04.9505696Z 2023-01-11T22:14:04.9505960Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9506069Z Ran 1 test in 6.454s 2023-01-11T22:14:04.9506088Z 2023-01-11T22:14:04.9506178Z OK 2023-01-11T22:14:04.9506197Z 2023-01-11T22:14:04.9506317Z Generating XML reports... 2023-01-11T22:14:04.9506769Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220028.xml 2023-01-11T22:14:04.9507137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9507294Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9507670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9507856Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9507922Z 2023-01-11T22:14:04.9508034Z Running tests... 2023-01-11T22:14:04.9508294Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9508604Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9508867Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9509616Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77325 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.622s) 2023-01-11T22:14:04.9509637Z 2023-01-11T22:14:04.9509894Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9509987Z Ran 1 test in 1.622s 2023-01-11T22:14:04.9510023Z 2023-01-11T22:14:04.9510113Z OK (skipped=1) 2023-01-11T22:14:04.9510135Z 2023-01-11T22:14:04.9510255Z Generating XML reports... 2023-01-11T22:14:04.9510698Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220037.xml 2023-01-11T22:14:04.9511065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9511239Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9511659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9511853Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9511872Z 2023-01-11T22:14:04.9511978Z Running tests... 2023-01-11T22:14:04.9512220Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9512531Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9512784Z test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9512999Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41957 2023-01-11T22:14:04.9513213Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41958 2023-01-11T22:14:04.9513580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9513755Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9514131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9514301Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9514839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9515010Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9515385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9515570Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9515811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9516055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9516450Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9516842Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9517049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9517274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9517433Z ok (6.513s) 2023-01-11T22:14:04.9517452Z 2023-01-11T22:14:04.9517716Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9517826Z Ran 1 test in 6.513s 2023-01-11T22:14:04.9517845Z 2023-01-11T22:14:04.9517935Z OK 2023-01-11T22:14:04.9517954Z 2023-01-11T22:14:04.9518076Z Generating XML reports... 2023-01-11T22:14:04.9518525Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220041.xml 2023-01-11T22:14:04.9518892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9519046Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9519409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9519599Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9519618Z 2023-01-11T22:14:04.9519724Z Running tests... 2023-01-11T22:14:04.9519983Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9520289Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9520555Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9520818Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42068 2023-01-11T22:14:04.9521019Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42069 2023-01-11T22:14:04.9521391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9521562Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9521936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9522124Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9522483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9522653Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9523023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9523208Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9523431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9523670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9524066Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9524463Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9524689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9524912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9525146Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9525376Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9525773Z /opt/conda/lib/python3.10/tempfile.py:860: ResourceWarning: Implicitly cleaning up 2023-01-11T22:14:04.9525919Z _warnings.warn(warn_message, ResourceWarning) 2023-01-11T22:14:04.9526310Z /opt/conda/lib/python3.10/tempfile.py:860: ResourceWarning: Implicitly cleaning up 2023-01-11T22:14:04.9526528Z _warnings.warn(warn_message, ResourceWarning) 2023-01-11T22:14:04.9526630Z ok (6.165s) 2023-01-11T22:14:04.9526649Z 2023-01-11T22:14:04.9526913Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9527024Z Ran 1 test in 6.165s 2023-01-11T22:14:04.9527043Z 2023-01-11T22:14:04.9527135Z OK 2023-01-11T22:14:04.9527154Z 2023-01-11T22:14:04.9527277Z Generating XML reports... 2023-01-11T22:14:04.9527709Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220050.xml 2023-01-11T22:14:04.9528076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9528251Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9528625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9528816Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9528835Z 2023-01-11T22:14:04.9528941Z Running tests... 2023-01-11T22:14:04.9529201Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9529509Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9529804Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9530028Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42183 2023-01-11T22:14:04.9530238Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42184 2023-01-11T22:14:04.9530791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9530964Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9531343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9531530Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9531888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9532057Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9532408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9532593Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9532833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9533072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9533469Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9533864Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9534088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9534310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9534542Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9534756Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9534858Z ok (4.354s) 2023-01-11T22:14:04.9534878Z 2023-01-11T22:14:04.9535139Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9535248Z Ran 1 test in 4.355s 2023-01-11T22:14:04.9535268Z 2023-01-11T22:14:04.9535358Z OK 2023-01-11T22:14:04.9535453Z 2023-01-11T22:14:04.9535583Z Generating XML reports... 2023-01-11T22:14:04.9536035Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220059.xml 2023-01-11T22:14:04.9536401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9536556Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9536933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9537119Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9537138Z 2023-01-11T22:14:04.9537244Z Running tests... 2023-01-11T22:14:04.9537504Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9537810Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9538073Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9538289Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42326 2023-01-11T22:14:04.9538485Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42327 2023-01-11T22:14:04.9538849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9539083Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9539471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9539657Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9540011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9540181Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9540556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9540739Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9540962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9541205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9541599Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9541991Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9542215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9542437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9542674Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9542903Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9543003Z ok (6.385s) 2023-01-11T22:14:04.9543022Z 2023-01-11T22:14:04.9543269Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9543386Z Ran 1 test in 6.385s 2023-01-11T22:14:04.9543406Z 2023-01-11T22:14:04.9543497Z OK 2023-01-11T22:14:04.9543516Z 2023-01-11T22:14:04.9543639Z Generating XML reports... 2023-01-11T22:14:04.9544087Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220106.xml 2023-01-11T22:14:04.9544498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9544674Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9545111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9545283Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9545319Z 2023-01-11T22:14:04.9545410Z Running tests... 2023-01-11T22:14:04.9545666Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9545980Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9546262Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9546476Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42441 2023-01-11T22:14:04.9546689Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42442 2023-01-11T22:14:04.9547055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9547230Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9547587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9547776Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9548179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9548358Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9548726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9548913Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9549154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9549397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9549774Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9550165Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9550393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9550615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9550849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T22:14:04.9551084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T22:14:04.9551479Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.9551869Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.9552107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2023-01-11T22:14:04.9552338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2023-01-11T22:14:04.9552706Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2023-01-11T22:14:04.9553094Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2023-01-11T22:14:04.9553195Z ok (5.720s) 2023-01-11T22:14:04.9553214Z 2023-01-11T22:14:04.9553477Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9553587Z Ran 1 test in 5.721s 2023-01-11T22:14:04.9553656Z 2023-01-11T22:14:04.9553750Z OK 2023-01-11T22:14:04.9553769Z 2023-01-11T22:14:04.9553890Z Generating XML reports... 2023-01-11T22:14:04.9554339Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220115.xml 2023-01-11T22:14:04.9554690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9554867Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9555242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9555429Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9555448Z 2023-01-11T22:14:04.9555554Z Running tests... 2023-01-11T22:14:04.9555813Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9556123Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9556401Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9556615Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42564 2023-01-11T22:14:04.9556812Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42565 2023-01-11T22:14:04.9557231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9557413Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9557791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9557977Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9558336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9558510Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9558882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9559049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9559289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9559530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9559925Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9560312Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9560536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9560763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9560997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T22:14:04.9561232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T22:14:04.9561612Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.9561996Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.9562231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2023-01-11T22:14:04.9562461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2023-01-11T22:14:04.9562843Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2023-01-11T22:14:04.9563302Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2023-01-11T22:14:04.9563404Z ok (15.876s) 2023-01-11T22:14:04.9563424Z 2023-01-11T22:14:04.9563685Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9563796Z Ran 1 test in 15.876s 2023-01-11T22:14:04.9563815Z 2023-01-11T22:14:04.9563893Z OK 2023-01-11T22:14:04.9563929Z 2023-01-11T22:14:04.9564035Z Generating XML reports... 2023-01-11T22:14:04.9564483Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220123.xml 2023-01-11T22:14:04.9564849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9565020Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9565396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9565586Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9565605Z 2023-01-11T22:14:04.9565711Z Running tests... 2023-01-11T22:14:04.9565969Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9566362Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9566673Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9566889Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42687 2023-01-11T22:14:04.9567101Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42688 2023-01-11T22:14:04.9567470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9567645Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9568021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9568207Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9568554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9568727Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9569095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9569279Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9569520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9569760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9570153Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9570721Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9570955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9571161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9571262Z ok (6.572s) 2023-01-11T22:14:04.9571282Z 2023-01-11T22:14:04.9571549Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9571658Z Ran 1 test in 6.572s 2023-01-11T22:14:04.9571678Z 2023-01-11T22:14:04.9571767Z OK 2023-01-11T22:14:04.9571785Z 2023-01-11T22:14:04.9571906Z Generating XML reports... 2023-01-11T22:14:04.9572443Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220141.xml 2023-01-11T22:14:04.9572809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9572983Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9573340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9573529Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9573548Z 2023-01-11T22:14:04.9573654Z Running tests... 2023-01-11T22:14:04.9573913Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9574218Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9574504Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9574721Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42802 2023-01-11T22:14:04.9574931Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42803 2023-01-11T22:14:04.9575281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9575513Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9575900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9576086Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9576445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9576616Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9576984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9577167Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9577410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9577633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9578027Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9578419Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9578643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9578866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9578972Z ok (6.409s) 2023-01-11T22:14:04.9578992Z 2023-01-11T22:14:04.9579254Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9579364Z Ran 1 test in 6.409s 2023-01-11T22:14:04.9579383Z 2023-01-11T22:14:04.9579457Z OK 2023-01-11T22:14:04.9579493Z 2023-01-11T22:14:04.9579600Z Generating XML reports... 2023-01-11T22:14:04.9580047Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220150.xml 2023-01-11T22:14:04.9580416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9580588Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9580962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9581149Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9581219Z 2023-01-11T22:14:04.9581329Z Running tests... 2023-01-11T22:14:04.9581592Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9581883Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9582138Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9582359Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42917 2023-01-11T22:14:04.9582571Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42918 2023-01-11T22:14:04.9582938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9583109Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9583480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9583669Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9584012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9584182Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9584548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9584777Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9585023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9585261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9585658Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9586051Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9586276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9586479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9586582Z ok (6.165s) 2023-01-11T22:14:04.9586601Z 2023-01-11T22:14:04.9586863Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9586973Z Ran 1 test in 6.165s 2023-01-11T22:14:04.9586993Z 2023-01-11T22:14:04.9587082Z OK 2023-01-11T22:14:04.9587101Z 2023-01-11T22:14:04.9587222Z Generating XML reports... 2023-01-11T22:14:04.9587670Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220159.xml 2023-01-11T22:14:04.9588037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9588212Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9588571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9588758Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9588777Z 2023-01-11T22:14:04.9588885Z Running tests... 2023-01-11T22:14:04.9589148Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9589462Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9589721Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9589933Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43028 2023-01-11T22:14:04.9590145Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43029 2023-01-11T22:14:04.9590558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9590731Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9591103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9591292Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9591656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9591826Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9592192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9592377Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9592622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9592846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9593242Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9593684Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9593920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9594142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9594914Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-01-11T22:14:04.9595686Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-01-11T22:14:04.9595923Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9596154Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9596259Z ok (6.231s) 2023-01-11T22:14:04.9596279Z 2023-01-11T22:14:04.9596552Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9596662Z Ran 1 test in 6.231s 2023-01-11T22:14:04.9596682Z 2023-01-11T22:14:04.9596756Z OK 2023-01-11T22:14:04.9596774Z 2023-01-11T22:14:04.9596898Z Generating XML reports... 2023-01-11T22:14:04.9597352Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220208.xml 2023-01-11T22:14:04.9597721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9597894Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9598267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9598510Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9598529Z 2023-01-11T22:14:04.9598636Z Running tests... 2023-01-11T22:14:04.9598879Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9599193Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9599469Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9600212Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78338 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.644s) 2023-01-11T22:14:04.9600233Z 2023-01-11T22:14:04.9600492Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9600599Z Ran 1 test in 1.644s 2023-01-11T22:14:04.9600622Z 2023-01-11T22:14:04.9600728Z OK (skipped=1) 2023-01-11T22:14:04.9600747Z 2023-01-11T22:14:04.9600869Z Generating XML reports... 2023-01-11T22:14:04.9601315Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220217.xml 2023-01-11T22:14:04.9601682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9601885Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9602271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9602458Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9602478Z 2023-01-11T22:14:04.9602585Z Running tests... 2023-01-11T22:14:04.9602846Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9603155Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9603435Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9604180Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77342 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.642s) 2023-01-11T22:14:04.9604200Z 2023-01-11T22:14:04.9604462Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9604573Z Ran 1 test in 1.643s 2023-01-11T22:14:04.9604592Z 2023-01-11T22:14:04.9604681Z OK (skipped=1) 2023-01-11T22:14:04.9604699Z 2023-01-11T22:14:04.9604821Z Generating XML reports... 2023-01-11T22:14:04.9605267Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220221.xml 2023-01-11T22:14:04.9605633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9605806Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9606181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9606367Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9606390Z 2023-01-11T22:14:04.9606498Z Running tests... 2023-01-11T22:14:04.9606742Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9607054Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9607330Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9607545Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43211 2023-01-11T22:14:04.9607813Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43212 2023-01-11T22:14:04.9608181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9608353Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9608729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9608916Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9609263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9609431Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9609796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9609981Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9610223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9610638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9611048Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9611518Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9611753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9611960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9612300Z STAGE:2023-01-11 22:02:31 43211:43211 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9612630Z STAGE:2023-01-11 22:02:31 43212:43212 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9612862Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9613093Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9613430Z STAGE:2023-01-11 22:02:32 43211:43211 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9613757Z STAGE:2023-01-11 22:02:32 43212:43212 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9614104Z STAGE:2023-01-11 22:02:32 43211:43211 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9614448Z STAGE:2023-01-11 22:02:32 43212:43212 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9615200Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-01-11T22:14:04.9615988Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-01-11T22:14:04.9616395Z STAGE:2023-01-11 22:02:32 43211:43211 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9616695Z STAGE:2023-01-11 22:02:32 43212:43212 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9617023Z STAGE:2023-01-11 22:02:32 43211:43211 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9617344Z STAGE:2023-01-11 22:02:32 43212:43212 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9617689Z STAGE:2023-01-11 22:02:32 43211:43211 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9618030Z STAGE:2023-01-11 22:02:32 43212:43212 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9618131Z ok (6.661s) 2023-01-11T22:14:04.9618152Z 2023-01-11T22:14:04.9618415Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9618529Z Ran 1 test in 6.661s 2023-01-11T22:14:04.9618548Z 2023-01-11T22:14:04.9618643Z OK 2023-01-11T22:14:04.9618662Z 2023-01-11T22:14:04.9618766Z Generating XML reports... 2023-01-11T22:14:04.9619214Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220225.xml 2023-01-11T22:14:04.9619581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9619754Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9620179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9620375Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9620394Z 2023-01-11T22:14:04.9620500Z Running tests... 2023-01-11T22:14:04.9620760Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9621053Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9621323Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9621539Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43330 2023-01-11T22:14:04.9621752Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43331 2023-01-11T22:14:04.9622122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9622294Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9622666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9622852Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9623212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9623366Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9623741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9623926Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9624167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9624409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9624807Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9625195Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9625418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9625709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9625794Z ok (5.789s) 2023-01-11T22:14:04.9625813Z 2023-01-11T22:14:04.9626075Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9626183Z Ran 1 test in 5.790s 2023-01-11T22:14:04.9626203Z 2023-01-11T22:14:04.9626292Z OK 2023-01-11T22:14:04.9626311Z 2023-01-11T22:14:04.9626434Z Generating XML reports... 2023-01-11T22:14:04.9626885Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220235.xml 2023-01-11T22:14:04.9627249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9627421Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9627779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9627973Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9627992Z 2023-01-11T22:14:04.9628098Z Running tests... 2023-01-11T22:14:04.9628357Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9628664Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9628987Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9629742Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78595 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.616s) 2023-01-11T22:14:04.9629762Z 2023-01-11T22:14:04.9630020Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9630134Z Ran 1 test in 1.617s 2023-01-11T22:14:04.9630154Z 2023-01-11T22:14:04.9630259Z OK (skipped=1) 2023-01-11T22:14:04.9630278Z 2023-01-11T22:14:04.9630383Z Generating XML reports... 2023-01-11T22:14:04.9630828Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220243.xml 2023-01-11T22:14:04.9631190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9631367Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9631742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9631932Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9631952Z 2023-01-11T22:14:04.9632059Z Running tests... 2023-01-11T22:14:04.9632317Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9632613Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9632889Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9633104Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43475 2023-01-11T22:14:04.9633318Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43476 2023-01-11T22:14:04.9633688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9633860Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9634235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9634421Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9634782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9634990Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9635364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9635550Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9635793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9636036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9636433Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9636824Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9637057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9637279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9638210Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2023-01-11T22:14:04.9638328Z warnings.warn( 2023-01-11T22:14:04.9639230Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2023-01-11T22:14:04.9639345Z warnings.warn( 2023-01-11T22:14:04.9639577Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9639801Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9639900Z ok (6.252s) 2023-01-11T22:14:04.9639920Z 2023-01-11T22:14:04.9640182Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9640300Z Ran 1 test in 6.252s 2023-01-11T22:14:04.9640320Z 2023-01-11T22:14:04.9640410Z OK 2023-01-11T22:14:04.9640430Z 2023-01-11T22:14:04.9640535Z Generating XML reports... 2023-01-11T22:14:04.9640983Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220247.xml 2023-01-11T22:14:04.9641347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9641522Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9641897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9642084Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9642103Z 2023-01-11T22:14:04.9642210Z Running tests... 2023-01-11T22:14:04.9642467Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9642763Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9643034Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9643772Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.649s) 2023-01-11T22:14:04.9643843Z 2023-01-11T22:14:04.9644108Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9644217Z Ran 1 test in 1.649s 2023-01-11T22:14:04.9644237Z 2023-01-11T22:14:04.9644342Z OK (skipped=1) 2023-01-11T22:14:04.9644361Z 2023-01-11T22:14:04.9644524Z Generating XML reports... 2023-01-11T22:14:04.9644978Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220256.xml 2023-01-11T22:14:04.9645343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9645516Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9645876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9646064Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9646087Z 2023-01-11T22:14:04.9646193Z Running tests... 2023-01-11T22:14:04.9646455Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9646765Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9647031Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9647296Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43624 2023-01-11T22:14:04.9647515Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43625 2023-01-11T22:14:04.9647887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9648042Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9648414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9648605Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9648964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9649133Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9649504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9649689Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9649929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9650152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9650789Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9651196Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9651419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9651641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9651977Z STAGE:2023-01-11 22:03:06 43624:43624 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9652298Z STAGE:2023-01-11 22:03:06 43625:43625 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9652529Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9652759Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:14:04.9653284Z STAGE:2023-01-11 22:03:06 43625:43625 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 22:03:06 43624:43624 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9653406Z 2023-01-11T22:14:04.9653750Z STAGE:2023-01-11 22:03:06 43625:43625 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9654094Z STAGE:2023-01-11 22:03:06 43624:43624 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9654420Z STAGE:2023-01-11 22:03:06 43624:43624 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9654751Z STAGE:2023-01-11 22:03:07 43624:43624 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9655087Z STAGE:2023-01-11 22:03:07 43624:43624 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9655189Z ok (6.956s) 2023-01-11T22:14:04.9655208Z 2023-01-11T22:14:04.9655471Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9655586Z Ran 1 test in 6.956s 2023-01-11T22:14:04.9655605Z 2023-01-11T22:14:04.9655679Z OK 2023-01-11T22:14:04.9655714Z 2023-01-11T22:14:04.9655818Z Generating XML reports... 2023-01-11T22:14:04.9656268Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220300.xml 2023-01-11T22:14:04.9656635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9656864Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9657257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9657445Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9657464Z 2023-01-11T22:14:04.9657571Z Running tests... 2023-01-11T22:14:04.9657830Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9658126Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9658388Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9658602Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43743 2023-01-11T22:14:04.9658814Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43744 2023-01-11T22:14:04.9659184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9659358Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9659731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9659918Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9660277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9660434Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9660801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9660986Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9661228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9661473Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9661869Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9662260Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9662485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9662747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9662850Z ok (5.764s) 2023-01-11T22:14:04.9662870Z 2023-01-11T22:14:04.9663133Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9663243Z Ran 1 test in 5.764s 2023-01-11T22:14:04.9663263Z 2023-01-11T22:14:04.9663354Z OK 2023-01-11T22:14:04.9663373Z 2023-01-11T22:14:04.9663497Z Generating XML reports... 2023-01-11T22:14:04.9663949Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220310.xml 2023-01-11T22:14:04.9664316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9664488Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9664846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9665037Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9665056Z 2023-01-11T22:14:04.9665162Z Running tests... 2023-01-11T22:14:04.9665422Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9665730Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9666048Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9666271Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43854 2023-01-11T22:14:04.9666485Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43855 2023-01-11T22:14:04.9666834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9667008Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9667386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9667572Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9667930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9668102Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9668471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9668654Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9668894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9669117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9669520Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9669912Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9670137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9670364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9670465Z ok (5.765s) 2023-01-11T22:14:04.9670485Z 2023-01-11T22:14:04.9670751Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9670861Z Ran 1 test in 5.765s 2023-01-11T22:14:04.9670881Z 2023-01-11T22:14:04.9670955Z OK 2023-01-11T22:14:04.9670989Z 2023-01-11T22:14:04.9671095Z Generating XML reports... 2023-01-11T22:14:04.9671540Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220318.xml 2023-01-11T22:14:04.9671964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9672136Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9672510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9672702Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9672721Z 2023-01-11T22:14:04.9672828Z Running tests... 2023-01-11T22:14:04.9673097Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9673391Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9673664Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9674406Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78684 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.658s) 2023-01-11T22:14:04.9674430Z 2023-01-11T22:14:04.9674686Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9674795Z Ran 1 test in 1.658s 2023-01-11T22:14:04.9674814Z 2023-01-11T22:14:04.9674978Z OK (skipped=1) 2023-01-11T22:14:04.9674998Z 2023-01-11T22:14:04.9675124Z Generating XML reports... 2023-01-11T22:14:04.9675571Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220326.xml 2023-01-11T22:14:04.9675936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9676110Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9676470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9676658Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9676678Z 2023-01-11T22:14:04.9676785Z Running tests... 2023-01-11T22:14:04.9677046Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9677359Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9677615Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9678357Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.616s) 2023-01-11T22:14:04.9678381Z 2023-01-11T22:14:04.9678645Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9678755Z Ran 1 test in 1.616s 2023-01-11T22:14:04.9678774Z 2023-01-11T22:14:04.9678865Z OK (skipped=1) 2023-01-11T22:14:04.9678902Z 2023-01-11T22:14:04.9679008Z Generating XML reports... 2023-01-11T22:14:04.9679452Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220330.xml 2023-01-11T22:14:04.9679821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9679991Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9680366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9680551Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9680570Z 2023-01-11T22:14:04.9680734Z Running tests... 2023-01-11T22:14:04.9680995Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9681289Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9681573Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9682314Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.647s) 2023-01-11T22:14:04.9682334Z 2023-01-11T22:14:04.9682593Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9682702Z Ran 1 test in 1.647s 2023-01-11T22:14:04.9682721Z 2023-01-11T22:14:04.9682827Z OK (skipped=1) 2023-01-11T22:14:04.9682849Z 2023-01-11T22:14:04.9682969Z Generating XML reports... 2023-01-11T22:14:04.9683411Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220335.xml 2023-01-11T22:14:04.9683779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9683950Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9684365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9684559Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9684578Z 2023-01-11T22:14:04.9684685Z Running tests... 2023-01-11T22:14:04.9684940Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9685251Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9685546Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9685759Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44067 2023-01-11T22:14:04.9685973Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44068 2023-01-11T22:14:04.9686322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9686497Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9686873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9687061Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9687419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9687587Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9687959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9688145Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9688385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9688612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9689010Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9689403Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9689629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9689855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9690025Z ok (6.260s) 2023-01-11T22:14:04.9690045Z 2023-01-11T22:14:04.9690480Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9690599Z Ran 1 test in 6.260s 2023-01-11T22:14:04.9690620Z 2023-01-11T22:14:04.9690696Z OK 2023-01-11T22:14:04.9690732Z 2023-01-11T22:14:04.9690838Z Generating XML reports... 2023-01-11T22:14:04.9691293Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220339.xml 2023-01-11T22:14:04.9691661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9691832Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9692206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9692397Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9692416Z 2023-01-11T22:14:04.9692522Z Running tests... 2023-01-11T22:14:04.9692783Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9693076Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9693341Z test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9693632Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44182 2023-01-11T22:14:04.9693855Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44183 2023-01-11T22:14:04.9694225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9694397Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9694771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9694959Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9695300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9695472Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9695842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9696027Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9696267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9696508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9696901Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9697294Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9697519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9697726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9698105Z /opt/conda/lib/python3.10/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op 2023-01-11T22:14:04.9698354Z warnings.warn("Initializing zero-element tensors is a no-op") 2023-01-11T22:14:04.9698726Z /opt/conda/lib/python3.10/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op 2023-01-11T22:14:04.9698973Z warnings.warn("Initializing zero-element tensors is a no-op") 2023-01-11T22:14:04.9699074Z ok (5.758s) 2023-01-11T22:14:04.9699094Z 2023-01-11T22:14:04.9699430Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9699541Z Ran 1 test in 5.758s 2023-01-11T22:14:04.9699562Z 2023-01-11T22:14:04.9699653Z OK 2023-01-11T22:14:04.9699672Z 2023-01-11T22:14:04.9699779Z Generating XML reports... 2023-01-11T22:14:04.9700230Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220348.xml 2023-01-11T22:14:04.9700594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9700767Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9701141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9701328Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9701347Z 2023-01-11T22:14:04.9701453Z Running tests... 2023-01-11T22:14:04.9701715Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9702004Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9702261Z test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9702474Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44293 2023-01-11T22:14:04.9702740Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44294 2023-01-11T22:14:04.9703116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9703288Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9703662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9703847Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9704216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9704371Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9704739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9704922Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9705168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9705410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9705805Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9706192Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9706420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9706645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9706861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T22:14:04.9707101Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T22:14:04.9707496Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.9707882Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.9707984Z ok (4.226s) 2023-01-11T22:14:04.9708003Z 2023-01-11T22:14:04.9708265Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9708481Z Ran 1 test in 4.226s 2023-01-11T22:14:04.9708500Z 2023-01-11T22:14:04.9708591Z OK 2023-01-11T22:14:04.9708610Z 2023-01-11T22:14:04.9708715Z Generating XML reports... 2023-01-11T22:14:04.9709166Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220356.xml 2023-01-11T22:14:04.9709531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9709707Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9710083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9710269Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9710289Z 2023-01-11T22:14:04.9710395Z Running tests... 2023-01-11T22:14:04.9710654Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9710965Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9711196Z test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9711408Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44408 2023-01-11T22:14:04.9711621Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44409 2023-01-11T22:14:04.9712031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9712211Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9712589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9712774Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9713135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9713292Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9713660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9713842Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9714087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9714327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9714721Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9715109Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9715333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9715558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9715774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T22:14:04.9716012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T22:14:04.9716408Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.9716796Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.9716895Z ok (4.198s) 2023-01-11T22:14:04.9716914Z 2023-01-11T22:14:04.9717175Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9717286Z Ran 1 test in 4.198s 2023-01-11T22:14:04.9717305Z 2023-01-11T22:14:04.9717448Z OK 2023-01-11T22:14:04.9717467Z 2023-01-11T22:14:04.9717589Z Generating XML reports... 2023-01-11T22:14:04.9718019Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220403.xml 2023-01-11T22:14:04.9718385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9718561Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9718939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9719128Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9719147Z 2023-01-11T22:14:04.9719254Z Running tests... 2023-01-11T22:14:04.9719514Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9719822Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9720080Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9720816Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78767 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.635s) 2023-01-11T22:14:04.9720853Z 2023-01-11T22:14:04.9721143Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9721261Z Ran 1 test in 1.635s 2023-01-11T22:14:04.9721280Z 2023-01-11T22:14:04.9721386Z OK (skipped=1) 2023-01-11T22:14:04.9721404Z 2023-01-11T22:14:04.9721526Z Generating XML reports... 2023-01-11T22:14:04.9721976Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220409.xml 2023-01-11T22:14:04.9722343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9722524Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9722899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9723069Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9723105Z 2023-01-11T22:14:04.9723195Z Running tests... 2023-01-11T22:14:04.9723462Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9723770Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9724040Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9724771Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78748 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.629s) 2023-01-11T22:14:04.9724794Z 2023-01-11T22:14:04.9725049Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9725157Z Ran 1 test in 1.629s 2023-01-11T22:14:04.9725176Z 2023-01-11T22:14:04.9725281Z OK (skipped=1) 2023-01-11T22:14:04.9725300Z 2023-01-11T22:14:04.9725425Z Generating XML reports... 2023-01-11T22:14:04.9725848Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220414.xml 2023-01-11T22:14:04.9726215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9726388Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9726760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9727003Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9727022Z 2023-01-11T22:14:04.9727128Z Running tests... 2023-01-11T22:14:04.9727391Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9727697Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9727951Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9728167Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44591 2023-01-11T22:14:04.9728378Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44592 2023-01-11T22:14:04.9728745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9728916Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9729294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9729481Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9729848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9730016Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9730607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9730813Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9731055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9731295Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9731701Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9732091Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9732316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9732543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9732628Z ok (4.244s) 2023-01-11T22:14:04.9732666Z 2023-01-11T22:14:04.9732908Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9733018Z Ran 1 test in 4.244s 2023-01-11T22:14:04.9733037Z 2023-01-11T22:14:04.9733127Z OK 2023-01-11T22:14:04.9733147Z 2023-01-11T22:14:04.9733269Z Generating XML reports... 2023-01-11T22:14:04.9733715Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220418.xml 2023-01-11T22:14:04.9734082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9734254Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9734632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9734806Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9734844Z 2023-01-11T22:14:04.9734935Z Running tests... 2023-01-11T22:14:04.9735197Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9735506Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9735743Z test_gather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9735959Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44700 2023-01-11T22:14:04.9736245Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44701 2023-01-11T22:14:04.9736614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9736770Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9737147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9737335Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9737692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9737861Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9738229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9738419Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9738657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9738894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9739275Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9739712Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9739944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9740167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9740502Z STAGE:2023-01-11 22:04:28 44700:44700 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9740828Z STAGE:2023-01-11 22:04:28 44701:44701 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9741159Z STAGE:2023-01-11 22:04:28 44701:44701 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9741506Z STAGE:2023-01-11 22:04:28 44701:44701 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9741842Z STAGE:2023-01-11 22:04:28 44700:44700 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9742167Z STAGE:2023-01-11 22:04:28 44700:44700 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9742492Z STAGE:2023-01-11 22:04:28 44701:44701 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9742807Z STAGE:2023-01-11 22:04:28 44700:44700 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9743246Z STAGE:2023-01-11 22:04:28 44700:44700 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9743596Z STAGE:2023-01-11 22:04:28 44700:44700 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9743923Z STAGE:2023-01-11 22:04:28 44701:44701 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9744262Z STAGE:2023-01-11 22:04:28 44701:44701 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9744363Z ok (4.358s) 2023-01-11T22:14:04.9744384Z 2023-01-11T22:14:04.9744683Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9744778Z Ran 1 test in 4.359s 2023-01-11T22:14:04.9744797Z 2023-01-11T22:14:04.9744892Z OK 2023-01-11T22:14:04.9744910Z 2023-01-11T22:14:04.9745032Z Generating XML reports... 2023-01-11T22:14:04.9745684Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220425.xml 2023-01-11T22:14:04.9746063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9746306Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9746690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9746876Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9746896Z 2023-01-11T22:14:04.9746986Z Running tests... 2023-01-11T22:14:04.9747252Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9747560Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9747808Z test_gather_checks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9748024Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44813 2023-01-11T22:14:04.9748237Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44814 2023-01-11T22:14:04.9748606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9748778Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9749153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9749375Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9749745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9749914Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9750278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9750464Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9750710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9750952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9751347Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9751722Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9751949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9752173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9752273Z ok (4.238s) 2023-01-11T22:14:04.9752293Z 2023-01-11T22:14:04.9752555Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9752666Z Ran 1 test in 4.239s 2023-01-11T22:14:04.9752688Z 2023-01-11T22:14:04.9752780Z OK 2023-01-11T22:14:04.9752799Z 2023-01-11T22:14:04.9752920Z Generating XML reports... 2023-01-11T22:14:04.9753420Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220431.xml 2023-01-11T22:14:04.9753769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9753947Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9754323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9754512Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9754531Z 2023-01-11T22:14:04.9754638Z Running tests... 2023-01-11T22:14:04.9754896Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9755205Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9755532Z test_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s) 2023-01-11T22:14:04.9755552Z 2023-01-11T22:14:04.9755812Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9755907Z Ran 1 test in 0.002s 2023-01-11T22:14:04.9755926Z 2023-01-11T22:14:04.9756032Z OK (skipped=1) 2023-01-11T22:14:04.9756051Z 2023-01-11T22:14:04.9756181Z Generating XML reports... 2023-01-11T22:14:04.9756629Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220438.xml 2023-01-11T22:14:04.9756995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9757168Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9757542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9757732Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9757750Z 2023-01-11T22:14:04.9757840Z Running tests... 2023-01-11T22:14:04.9758102Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9758411Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9758714Z test_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9758936Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44955 2023-01-11T22:14:04.9759150Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44956 2023-01-11T22:14:04.9759514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9759687Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9760065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9760234Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9760590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9760762Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9761135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9761321Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9761561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9761801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9762201Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9762572Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9762800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9763025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9763263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T22:14:04.9763499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T22:14:04.9763890Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.9764272Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.9764662Z STAGE:2023-01-11 22:04:45 44955:44955 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9764978Z STAGE:2023-01-11 22:04:45 44956:44956 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9765311Z STAGE:2023-01-11 22:04:45 44956:44956 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9765642Z STAGE:2023-01-11 22:04:45 44956:44956 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9765970Z STAGE:2023-01-11 22:04:45 44955:44955 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9766310Z STAGE:2023-01-11 22:04:45 44955:44955 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9766635Z STAGE:2023-01-11 22:04:45 44956:44956 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9766955Z STAGE:2023-01-11 22:04:45 44955:44955 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9767288Z STAGE:2023-01-11 22:04:45 44955:44955 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9767629Z STAGE:2023-01-11 22:04:45 44955:44955 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9767957Z STAGE:2023-01-11 22:04:45 44956:44956 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9768326Z STAGE:2023-01-11 22:04:45 44956:44956 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9768434Z ok (4.458s) 2023-01-11T22:14:04.9768454Z 2023-01-11T22:14:04.9768717Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9768828Z Ran 1 test in 4.458s 2023-01-11T22:14:04.9768847Z 2023-01-11T22:14:04.9768939Z OK 2023-01-11T22:14:04.9768958Z 2023-01-11T22:14:04.9769079Z Generating XML reports... 2023-01-11T22:14:04.9769531Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220441.xml 2023-01-11T22:14:04.9769897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9770071Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9770613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9770816Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9770836Z 2023-01-11T22:14:04.9770947Z Running tests... 2023-01-11T22:14:04.9771207Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9771517Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9771766Z test_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9771985Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45074 2023-01-11T22:14:04.9772189Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45075 2023-01-11T22:14:04.9772540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9772712Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9773090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9773277Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9773638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9773809Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9774177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9774450Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9774694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9774917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9775320Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9775712Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9775936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9776161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9776317Z skip: Skipped due to small world size. (4.242s) 2023-01-11T22:14:04.9776340Z 2023-01-11T22:14:04.9776607Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9776717Z Ran 1 test in 4.242s 2023-01-11T22:14:04.9776737Z 2023-01-11T22:14:04.9776841Z OK (skipped=1) 2023-01-11T22:14:04.9776860Z 2023-01-11T22:14:04.9776964Z Generating XML reports... 2023-01-11T22:14:04.9777475Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220448.xml 2023-01-11T22:14:04.9777856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9778025Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9778401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9778587Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9778611Z 2023-01-11T22:14:04.9778716Z Running tests... 2023-01-11T22:14:04.9778976Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9779267Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9779515Z test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9779732Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45183 2023-01-11T22:14:04.9779945Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45184 2023-01-11T22:14:04.9780306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9780477Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9780850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9781040Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9781398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9781552Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9781920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9782106Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9782346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9782586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9782980Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9783432Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9783657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9783880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9783964Z ok (4.251s) 2023-01-11T22:14:04.9783984Z 2023-01-11T22:14:04.9784249Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9784359Z Ran 1 test in 4.251s 2023-01-11T22:14:04.9784378Z 2023-01-11T22:14:04.9784468Z OK 2023-01-11T22:14:04.9784487Z 2023-01-11T22:14:04.9784611Z Generating XML reports... 2023-01-11T22:14:04.9785058Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220455.xml 2023-01-11T22:14:04.9785424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9785599Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9785955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9786142Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9786161Z 2023-01-11T22:14:04.9786267Z Running tests... 2023-01-11T22:14:04.9786576Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9786896Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9787159Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9787900Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.662s) 2023-01-11T22:14:04.9787923Z 2023-01-11T22:14:04.9788181Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9788291Z Ran 1 test in 1.662s 2023-01-11T22:14:04.9788310Z 2023-01-11T22:14:04.9788415Z OK (skipped=1) 2023-01-11T22:14:04.9788435Z 2023-01-11T22:14:04.9788539Z Generating XML reports... 2023-01-11T22:14:04.9788988Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220501.xml 2023-01-11T22:14:04.9789357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9789529Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9789900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9790092Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9790112Z 2023-01-11T22:14:04.9790219Z Running tests... 2023-01-11T22:14:04.9790479Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9790772Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9791016Z test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9791233Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45326 2023-01-11T22:14:04.9791445Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45327 2023-01-11T22:14:04.9791812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9791983Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9792354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9792595Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9792957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9793108Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9793479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9793663Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9793903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9794142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9794536Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9794932Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9795155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9795363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9795644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T22:14:04.9795888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T22:14:04.9796281Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.9796668Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.9796773Z ok (4.215s) 2023-01-11T22:14:04.9796793Z 2023-01-11T22:14:04.9797054Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9797163Z Ran 1 test in 4.216s 2023-01-11T22:14:04.9797183Z 2023-01-11T22:14:04.9797274Z OK 2023-01-11T22:14:04.9797293Z 2023-01-11T22:14:04.9797398Z Generating XML reports... 2023-01-11T22:14:04.9797844Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220506.xml 2023-01-11T22:14:04.9798213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9798386Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9798757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9798945Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9798967Z 2023-01-11T22:14:04.9799073Z Running tests... 2023-01-11T22:14:04.9799331Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9799640Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9799864Z test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9800077Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45441 2023-01-11T22:14:04.9800292Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45442 2023-01-11T22:14:04.9800660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9800831Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9801203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9801444Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9801808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9801959Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9802335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9802522Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9802761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9803003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9803398Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9803786Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9804016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9804238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9804321Z ok (4.169s) 2023-01-11T22:14:04.9804340Z 2023-01-11T22:14:04.9804605Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9804764Z Ran 1 test in 4.169s 2023-01-11T22:14:04.9804784Z 2023-01-11T22:14:04.9804878Z OK 2023-01-11T22:14:04.9804897Z 2023-01-11T22:14:04.9805021Z Generating XML reports... 2023-01-11T22:14:04.9805469Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220512.xml 2023-01-11T22:14:04.9805838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9806014Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9806371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9806558Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9806577Z 2023-01-11T22:14:04.9806685Z Running tests... 2023-01-11T22:14:04.9806943Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9807257Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9807494Z test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9807710Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45550 2023-01-11T22:14:04.9807925Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45551 2023-01-11T22:14:04.9808290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9808449Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9808824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9809012Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9809372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9809544Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9809914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9810097Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9810508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9810824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9811229Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9811621Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9811852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9812077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9812178Z ok (4.147s) 2023-01-11T22:14:04.9812199Z 2023-01-11T22:14:04.9812469Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9812579Z Ran 1 test in 4.147s 2023-01-11T22:14:04.9812599Z 2023-01-11T22:14:04.9812689Z OK 2023-01-11T22:14:04.9812708Z 2023-01-11T22:14:04.9812816Z Generating XML reports... 2023-01-11T22:14:04.9813263Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220519.xml 2023-01-11T22:14:04.9813629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9813803Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9814235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9814431Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9814450Z 2023-01-11T22:14:04.9814557Z Running tests... 2023-01-11T22:14:04.9814820Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9815134Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9815380Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9815600Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45659 2023-01-11T22:14:04.9815811Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45660 2023-01-11T22:14:04.9816178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9816352Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9816722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9816907Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9817264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9817415Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9817784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9817967Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9818207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9818448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9818848Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9819238Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9819462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9819685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9819972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T22:14:04.9820208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T22:14:04.9820603Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.9820992Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.9821094Z ok (4.259s) 2023-01-11T22:14:04.9821114Z 2023-01-11T22:14:04.9821373Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9821483Z Ran 1 test in 4.260s 2023-01-11T22:14:04.9821503Z 2023-01-11T22:14:04.9821593Z OK 2023-01-11T22:14:04.9821612Z 2023-01-11T22:14:04.9821732Z Generating XML reports... 2023-01-11T22:14:04.9822162Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220526.xml 2023-01-11T22:14:04.9822532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9822705Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9823079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9823316Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9823337Z 2023-01-11T22:14:04.9823449Z Running tests... 2023-01-11T22:14:04.9823710Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9824020Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9824259Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9824477Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45774 2023-01-11T22:14:04.9824689Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45775 2023-01-11T22:14:04.9825057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9825227Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9825602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9825792Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9826150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9826318Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9826669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9826858Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9827099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9827339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9827736Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9828128Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9828353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9828575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9828810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T22:14:04.9829086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T22:14:04.9829482Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.9829869Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.9829977Z ok (4.255s) 2023-01-11T22:14:04.9829997Z 2023-01-11T22:14:04.9830256Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9830366Z Ran 1 test in 4.255s 2023-01-11T22:14:04.9830386Z 2023-01-11T22:14:04.9830477Z OK 2023-01-11T22:14:04.9830496Z 2023-01-11T22:14:04.9830618Z Generating XML reports... 2023-01-11T22:14:04.9831048Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220533.xml 2023-01-11T22:14:04.9831421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9831592Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9831972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9832158Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9832226Z 2023-01-11T22:14:04.9832341Z Running tests... 2023-01-11T22:14:04.9832603Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9832917Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9833175Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9833372Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45889 2023-01-11T22:14:04.9833589Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45890 2023-01-11T22:14:04.9833955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9834127Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9834502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9834696Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9835056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9835224Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9835577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9835765Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9836007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9836246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9836641Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9837035Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9837261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9837485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9837587Z ok (6.161s) 2023-01-11T22:14:04.9837607Z 2023-01-11T22:14:04.9837850Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9838016Z Ran 1 test in 6.162s 2023-01-11T22:14:04.9838035Z 2023-01-11T22:14:04.9838126Z OK 2023-01-11T22:14:04.9838145Z 2023-01-11T22:14:04.9838268Z Generating XML reports... 2023-01-11T22:14:04.9838718Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220539.xml 2023-01-11T22:14:04.9839084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9839255Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9839629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9839816Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9839836Z 2023-01-11T22:14:04.9839926Z Running tests... 2023-01-11T22:14:04.9840185Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9840499Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9840732Z test_irecv (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9840947Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46004 2023-01-11T22:14:04.9841160Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46005 2023-01-11T22:14:04.9841569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9841746Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9842104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9842291Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9842648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9842823Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9843191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9843374Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9843617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9843855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9844233Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9844666Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9844896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9845118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9845218Z ok (4.175s) 2023-01-11T22:14:04.9845237Z 2023-01-11T22:14:04.9845501Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9845610Z Ran 1 test in 4.175s 2023-01-11T22:14:04.9845629Z 2023-01-11T22:14:04.9845724Z OK 2023-01-11T22:14:04.9845746Z 2023-01-11T22:14:04.9845868Z Generating XML reports... 2023-01-11T22:14:04.9846299Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220548.xml 2023-01-11T22:14:04.9846667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9846841Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9847215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9847469Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9847489Z 2023-01-11T22:14:04.9847596Z Running tests... 2023-01-11T22:14:04.9847859Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9848167Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9848405Z test_isend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9848603Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46113 2023-01-11T22:14:04.9848816Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46114 2023-01-11T22:14:04.9849182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9849356Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9849731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9849918Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9850437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9850751Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9851123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9851310Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9851556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9851954Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9852191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9852576Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9852801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9853024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9853125Z ok (4.243s) 2023-01-11T22:14:04.9853145Z 2023-01-11T22:14:04.9853388Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9853499Z Ran 1 test in 4.243s 2023-01-11T22:14:04.9853518Z 2023-01-11T22:14:04.9853609Z OK 2023-01-11T22:14:04.9853628Z 2023-01-11T22:14:04.9853748Z Generating XML reports... 2023-01-11T22:14:04.9854197Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220555.xml 2023-01-11T22:14:04.9854569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9854742Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9855117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9855293Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9855329Z 2023-01-11T22:14:04.9855421Z Running tests... 2023-01-11T22:14:04.9855683Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9855995Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9856261Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9856553Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46222 2023-01-11T22:14:04.9856765Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46223 2023-01-11T22:14:04.9857133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9857306Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9857664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9857851Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9858206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9858373Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9858736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9858925Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9859167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9859406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9859830Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9860229Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9860454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9860677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9861006Z STAGE:2023-01-11 22:06:05 46223:46223 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9861329Z STAGE:2023-01-11 22:06:05 46222:46222 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9861656Z STAGE:2023-01-11 22:06:05 46222:46222 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9861999Z STAGE:2023-01-11 22:06:05 46222:46222 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9862335Z STAGE:2023-01-11 22:06:05 46223:46223 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9862660Z STAGE:2023-01-11 22:06:05 46223:46223 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9862763Z ok (4.254s) 2023-01-11T22:14:04.9862783Z 2023-01-11T22:14:04.9863044Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9863153Z Ran 1 test in 4.254s 2023-01-11T22:14:04.9863172Z 2023-01-11T22:14:04.9863264Z OK 2023-01-11T22:14:04.9863286Z 2023-01-11T22:14:04.9863408Z Generating XML reports... 2023-01-11T22:14:04.9863851Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220602.xml 2023-01-11T22:14:04.9864217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9864388Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9864746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9864936Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9864956Z 2023-01-11T22:14:04.9865061Z Running tests... 2023-01-11T22:14:04.9865318Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9865627Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9865945Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9866162Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46335 2023-01-11T22:14:04.9866374Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46336 2023-01-11T22:14:04.9866723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9866898Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9867272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9867459Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9867814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9867987Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9868362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9868545Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9868786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9869053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9869459Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9869848Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9870072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9870295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9870633Z STAGE:2023-01-11 22:06:12 46335:46335 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9870952Z STAGE:2023-01-11 22:06:12 46336:46336 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:04.9871282Z STAGE:2023-01-11 22:06:12 46335:46335 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9871629Z STAGE:2023-01-11 22:06:12 46335:46335 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9871943Z STAGE:2023-01-11 22:06:12 46336:46336 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:04.9872279Z STAGE:2023-01-11 22:06:12 46336:46336 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:04.9872379Z ok (4.230s) 2023-01-11T22:14:04.9872399Z 2023-01-11T22:14:04.9872660Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9872774Z Ran 1 test in 4.230s 2023-01-11T22:14:04.9872793Z 2023-01-11T22:14:04.9872883Z OK 2023-01-11T22:14:04.9872902Z 2023-01-11T22:14:04.9873022Z Generating XML reports... 2023-01-11T22:14:04.9873471Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220608.xml 2023-01-11T22:14:04.9873816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9873993Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9874364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9874550Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9874569Z 2023-01-11T22:14:04.9874674Z Running tests... 2023-01-11T22:14:04.9874933Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9875301Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9875581Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9875795Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46448 2023-01-11T22:14:04.9875991Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46449 2023-01-11T22:14:04.9876360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9876530Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9876903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9877088Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9877445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9877616Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9877981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9878149Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9878436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9878679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9879073Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9879460Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9879686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9879909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9880143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T22:14:04.9880380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T22:14:04.9880756Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.9881147Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.9881381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2023-01-11T22:14:04.9881612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2023-01-11T22:14:04.9882005Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2023-01-11T22:14:04.9882389Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2023-01-11T22:14:04.9882616Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms 2023-01-11T22:14:04.9882717Z ok (22.355s) 2023-01-11T22:14:04.9882736Z 2023-01-11T22:14:04.9883004Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9883099Z Ran 1 test in 22.355s 2023-01-11T22:14:04.9883135Z 2023-01-11T22:14:04.9883208Z OK 2023-01-11T22:14:04.9883227Z 2023-01-11T22:14:04.9883347Z Generating XML reports... 2023-01-11T22:14:04.9883795Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220615.xml 2023-01-11T22:14:04.9884162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9884396Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9884771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9884956Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9884975Z 2023-01-11T22:14:04.9885082Z Running tests... 2023-01-11T22:14:04.9885327Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9885635Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9885927Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9886141Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46575 2023-01-11T22:14:04.9886354Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46576 2023-01-11T22:14:04.9886724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9886893Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9887265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9887483Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9887850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9888018Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9888385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9888568Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9888812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9889049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9889441Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9889831Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9890039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9890262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9890671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T22:14:04.9890905Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T22:14:04.9891306Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.9891694Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.9891924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2023-01-11T22:14:04.9892158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2023-01-11T22:14:04.9892542Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2023-01-11T22:14:04.9892911Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2023-01-11T22:14:04.9893140Z [E ProcessGroupGloo.cpp:2803] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms 2023-01-11T22:14:04.9893452Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 100 ms 2023-01-11T22:14:04.9893552Z ok (22.880s) 2023-01-11T22:14:04.9893572Z 2023-01-11T22:14:04.9893836Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9893948Z Ran 1 test in 22.881s 2023-01-11T22:14:04.9893968Z 2023-01-11T22:14:04.9894058Z OK 2023-01-11T22:14:04.9894077Z 2023-01-11T22:14:04.9894199Z Generating XML reports... 2023-01-11T22:14:04.9894647Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220640.xml 2023-01-11T22:14:04.9894999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9895173Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9895548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9895739Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9895758Z 2023-01-11T22:14:04.9895865Z Running tests... 2023-01-11T22:14:04.9896124Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9896433Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9896768Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9896975Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46702 2023-01-11T22:14:04.9897189Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46703 2023-01-11T22:14:04.9897558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9897732Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9898110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9898295Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9898649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9898815Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9899183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9899351Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9899589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9899977Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9900214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9900602Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9900823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9901048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9901201Z skip: Skipped due to small world size. (4.108s) 2023-01-11T22:14:04.9901221Z 2023-01-11T22:14:04.9901480Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9901573Z Ran 1 test in 4.108s 2023-01-11T22:14:04.9901592Z 2023-01-11T22:14:04.9901699Z OK (skipped=1) 2023-01-11T22:14:04.9901717Z 2023-01-11T22:14:04.9901838Z Generating XML reports... 2023-01-11T22:14:04.9902286Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220705.xml 2023-01-11T22:14:04.9902711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9902882Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9903256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9903444Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9903463Z 2023-01-11T22:14:04.9903569Z Running tests... 2023-01-11T22:14:04.9903814Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9904121Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9904379Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9904594Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46811 2023-01-11T22:14:04.9904805Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46812 2023-01-11T22:14:04.9905172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9905338Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9905756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9905932Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9906294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9906460Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9906829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9907016Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9907255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9907494Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9907887Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9908278Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9908486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9908706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9908931Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms 2023-01-11T22:14:04.9909033Z ok (6.314s) 2023-01-11T22:14:04.9909052Z 2023-01-11T22:14:04.9909309Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9909414Z Ran 1 test in 6.314s 2023-01-11T22:14:04.9909434Z 2023-01-11T22:14:04.9909524Z OK 2023-01-11T22:14:04.9909542Z 2023-01-11T22:14:04.9909664Z Generating XML reports... 2023-01-11T22:14:04.9910094Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220712.xml 2023-01-11T22:14:04.9910460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9910631Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9911001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9911185Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9911254Z 2023-01-11T22:14:04.9911366Z Running tests... 2023-01-11T22:14:04.9911628Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9911937Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9912214Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9912415Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46920 2023-01-11T22:14:04.9912628Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46921 2023-01-11T22:14:04.9912992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9913162Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9913534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9913726Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9914086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9914256Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9914652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9914842Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9915083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9915323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9915723Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9916121Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9916346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9916569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9916802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T22:14:04.9917019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T22:14:04.9917412Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.9917798Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.9918013Z [E ProcessGroupGloo.cpp:138] Rank 0 timed out in monitoredBarrier after 0 ms. 2023-01-11T22:14:04.9918186Z No ranks successfully processed in monitoredBarrier. 2023-01-11T22:14:04.9918403Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 0 ms 2023-01-11T22:14:04.9918503Z ok (4.337s) 2023-01-11T22:14:04.9918523Z 2023-01-11T22:14:04.9918785Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9918878Z Ran 1 test in 4.337s 2023-01-11T22:14:04.9918913Z 2023-01-11T22:14:04.9918991Z OK 2023-01-11T22:14:04.9919009Z 2023-01-11T22:14:04.9919131Z Generating XML reports... 2023-01-11T22:14:04.9919575Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220721.xml 2023-01-11T22:14:04.9919943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9920114Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9920546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9920733Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9920753Z 2023-01-11T22:14:04.9920859Z Running tests... 2023-01-11T22:14:04.9921099Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9921409Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9921679Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9921892Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47035 2023-01-11T22:14:04.9922099Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47036 2023-01-11T22:14:04.9922462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9922634Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9923006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9923176Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9923580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9923752Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9924125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9924308Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9924547Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9924788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9925175Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9925562Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9925772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9925995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9926222Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T22:14:04.9926452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T22:14:04.9926841Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.9927231Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:04.9927448Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms 2023-01-11T22:14:04.9927544Z ok (4.453s) 2023-01-11T22:14:04.9927564Z 2023-01-11T22:14:04.9927825Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9927921Z Ran 1 test in 4.453s 2023-01-11T22:14:04.9927940Z 2023-01-11T22:14:04.9928031Z OK 2023-01-11T22:14:04.9928049Z 2023-01-11T22:14:04.9928169Z Generating XML reports... 2023-01-11T22:14:04.9928612Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220728.xml 2023-01-11T22:14:04.9928976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9929145Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9929576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9929761Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9929781Z 2023-01-11T22:14:04.9929886Z Running tests... 2023-01-11T22:14:04.9930126Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9930606Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9930892Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9931105Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47150 2023-01-11T22:14:04.9931317Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47151 2023-01-11T22:14:04.9931688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9931863Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9932237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9932407Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9932835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9933014Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9933389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9933670Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9933913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9934155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9934554Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9934942Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9935154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9935377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9935531Z skip: Skipped due to small world size. (4.254s) 2023-01-11T22:14:04.9935550Z 2023-01-11T22:14:04.9935814Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9935924Z Ran 1 test in 4.255s 2023-01-11T22:14:04.9935943Z 2023-01-11T22:14:04.9936047Z OK (skipped=1) 2023-01-11T22:14:04.9936069Z 2023-01-11T22:14:04.9936189Z Generating XML reports... 2023-01-11T22:14:04.9936633Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220735.xml 2023-01-11T22:14:04.9936998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9937157Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9937530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9937771Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9937802Z 2023-01-11T22:14:04.9937993Z Running tests... 2023-01-11T22:14:04.9938263Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9938572Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9939061Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s) 2023-01-11T22:14:04.9939082Z 2023-01-11T22:14:04.9939339Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9939432Z Ran 1 test in 0.002s 2023-01-11T22:14:04.9939468Z 2023-01-11T22:14:04.9939558Z OK (skipped=1) 2023-01-11T22:14:04.9939577Z 2023-01-11T22:14:04.9939699Z Generating XML reports... 2023-01-11T22:14:04.9940143Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220742.xml 2023-01-11T22:14:04.9940506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9940678Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9941053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9941240Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9941259Z 2023-01-11T22:14:04.9941362Z Running tests... 2023-01-11T22:14:04.9941602Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9941909Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9942349Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s) 2023-01-11T22:14:04.9942370Z 2023-01-11T22:14:04.9942630Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9942739Z Ran 1 test in 0.002s 2023-01-11T22:14:04.9942758Z 2023-01-11T22:14:04.9942862Z OK (skipped=1) 2023-01-11T22:14:04.9942880Z 2023-01-11T22:14:04.9943001Z Generating XML reports... 2023-01-11T22:14:04.9943437Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220744.xml 2023-01-11T22:14:04.9943804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9943961Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9944331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9944563Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9944582Z 2023-01-11T22:14:04.9944691Z Running tests... 2023-01-11T22:14:04.9944951Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9945255Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9945647Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s) 2023-01-11T22:14:04.9945670Z 2023-01-11T22:14:04.9945923Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9946028Z Ran 1 test in 0.002s 2023-01-11T22:14:04.9946047Z 2023-01-11T22:14:04.9946138Z OK (skipped=1) 2023-01-11T22:14:04.9946157Z 2023-01-11T22:14:04.9946278Z Generating XML reports... 2023-01-11T22:14:04.9946710Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220747.xml 2023-01-11T22:14:04.9947072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9947242Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9947608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9947791Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9947810Z 2023-01-11T22:14:04.9947983Z Running tests... 2023-01-11T22:14:04.9948226Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9948535Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9948924Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.003s) 2023-01-11T22:14:04.9948943Z 2023-01-11T22:14:04.9949196Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9949306Z Ran 1 test in 0.003s 2023-01-11T22:14:04.9949324Z 2023-01-11T22:14:04.9949429Z OK (skipped=1) 2023-01-11T22:14:04.9949447Z 2023-01-11T22:14:04.9949566Z Generating XML reports... 2023-01-11T22:14:04.9950001Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220749.xml 2023-01-11T22:14:04.9950363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9950521Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9950896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9951079Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9951098Z 2023-01-11T22:14:04.9951202Z Running tests... 2023-01-11T22:14:04.9951509Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9951823Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9952116Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL backend supports high priority stream (0.002s) 2023-01-11T22:14:04.9952136Z 2023-01-11T22:14:04.9952392Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9952499Z Ran 1 test in 0.002s 2023-01-11T22:14:04.9952522Z 2023-01-11T22:14:04.9952610Z OK (skipped=1) 2023-01-11T22:14:04.9952629Z 2023-01-11T22:14:04.9952754Z Generating XML reports... 2023-01-11T22:14:04.9953194Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220751.xml 2023-01-11T22:14:04.9953559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9953728Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9954104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9954292Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9954311Z 2023-01-11T22:14:04.9954416Z Running tests... 2023-01-11T22:14:04.9954675Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9954966Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9955214Z test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2023-01-11T22:14:04.9955234Z 2023-01-11T22:14:04.9955488Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9955595Z Ran 1 test in 0.002s 2023-01-11T22:14:04.9955614Z 2023-01-11T22:14:04.9955716Z OK (skipped=1) 2023-01-11T22:14:04.9955735Z 2023-01-11T22:14:04.9955853Z Generating XML reports... 2023-01-11T22:14:04.9956295Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220754.xml 2023-01-11T22:14:04.9956659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9956814Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9957184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9957427Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9957446Z 2023-01-11T22:14:04.9957550Z Running tests... 2023-01-11T22:14:04.9957806Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9958112Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9958381Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2023-01-11T22:14:04.9958400Z 2023-01-11T22:14:04.9958654Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9958762Z Ran 1 test in 0.002s 2023-01-11T22:14:04.9958781Z 2023-01-11T22:14:04.9958871Z OK (skipped=1) 2023-01-11T22:14:04.9958904Z 2023-01-11T22:14:04.9959009Z Generating XML reports... 2023-01-11T22:14:04.9959452Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220756.xml 2023-01-11T22:14:04.9959821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9959992Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9960367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9960603Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9960623Z 2023-01-11T22:14:04.9960734Z Running tests... 2023-01-11T22:14:04.9960996Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9961289Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9961592Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2023-01-11T22:14:04.9961615Z 2023-01-11T22:14:04.9961867Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9961973Z Ran 1 test in 0.002s 2023-01-11T22:14:04.9961993Z 2023-01-11T22:14:04.9962094Z OK (skipped=1) 2023-01-11T22:14:04.9962113Z 2023-01-11T22:14:04.9962231Z Generating XML reports... 2023-01-11T22:14:04.9962670Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220759.xml 2023-01-11T22:14:04.9963040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9963209Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9963567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9963749Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9963768Z 2023-01-11T22:14:04.9963880Z Running tests... 2023-01-11T22:14:04.9964139Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9964444Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9964740Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9964960Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47523 2023-01-11T22:14:04.9965171Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47524 2023-01-11T22:14:04.9965519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9965688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9966058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9966299Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9966656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9966823Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9967188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9967374Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9967619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9967842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9968234Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9968631Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9968858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9969079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9969178Z ok (4.291s) 2023-01-11T22:14:04.9969197Z 2023-01-11T22:14:04.9969503Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9969616Z Ran 1 test in 4.291s 2023-01-11T22:14:04.9969635Z 2023-01-11T22:14:04.9969708Z OK 2023-01-11T22:14:04.9969743Z 2023-01-11T22:14:04.9969848Z Generating XML reports... 2023-01-11T22:14:04.9970461Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220801.xml 2023-01-11T22:14:04.9970849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9971027Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9971398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9971583Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9971602Z 2023-01-11T22:14:04.9971707Z Running tests... 2023-01-11T22:14:04.9971966Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9972255Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9972543Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9972755Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47632 2023-01-11T22:14:04.9972966Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47633 2023-01-11T22:14:04.9973339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9973510Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9973880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9974066Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9974430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9974582Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9974945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9975125Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9975364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9975686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9976082Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9976475Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9976704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9976911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9977011Z ok (4.231s) 2023-01-11T22:14:04.9977031Z 2023-01-11T22:14:04.9977289Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9977397Z Ran 1 test in 4.231s 2023-01-11T22:14:04.9977416Z 2023-01-11T22:14:04.9977508Z OK 2023-01-11T22:14:04.9977527Z 2023-01-11T22:14:04.9977645Z Generating XML reports... 2023-01-11T22:14:04.9978089Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220808.xml 2023-01-11T22:14:04.9978453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9978622Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9979038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9979233Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9979252Z 2023-01-11T22:14:04.9979358Z Running tests... 2023-01-11T22:14:04.9979616Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9979923Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9980198Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2023-01-11T22:14:04.9980217Z 2023-01-11T22:14:04.9980475Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9980585Z Ran 1 test in 0.002s 2023-01-11T22:14:04.9980604Z 2023-01-11T22:14:04.9980709Z OK (skipped=1) 2023-01-11T22:14:04.9980728Z 2023-01-11T22:14:04.9980831Z Generating XML reports... 2023-01-11T22:14:04.9981277Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220815.xml 2023-01-11T22:14:04.9981641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9981812Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9982185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9982377Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9982396Z 2023-01-11T22:14:04.9982503Z Running tests... 2023-01-11T22:14:04.9982758Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9983046Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9983343Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2023-01-11T22:14:04.9983363Z 2023-01-11T22:14:04.9983616Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9983726Z Ran 1 test in 0.002s 2023-01-11T22:14:04.9983746Z 2023-01-11T22:14:04.9983850Z OK (skipped=1) 2023-01-11T22:14:04.9983869Z 2023-01-11T22:14:04.9983986Z Generating XML reports... 2023-01-11T22:14:04.9984423Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220817.xml 2023-01-11T22:14:04.9984851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9985022Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9985378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9985567Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9985586Z 2023-01-11T22:14:04.9985693Z Running tests... 2023-01-11T22:14:04.9985952Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9986261Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9986534Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9987277Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.667s) 2023-01-11T22:14:04.9987298Z 2023-01-11T22:14:04.9987554Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9987663Z Ran 1 test in 1.668s 2023-01-11T22:14:04.9987729Z 2023-01-11T22:14:04.9987835Z OK (skipped=1) 2023-01-11T22:14:04.9987854Z 2023-01-11T22:14:04.9987960Z Generating XML reports... 2023-01-11T22:14:04.9988401Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220819.xml 2023-01-11T22:14:04.9988762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9988930Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9989311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9989497Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9989516Z 2023-01-11T22:14:04.9989621Z Running tests... 2023-01-11T22:14:04.9989883Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9990177Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9990454Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9990669Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47841 2023-01-11T22:14:04.9990879Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47842 2023-01-11T22:14:04.9991241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9991414Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9991788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9991973Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9992332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9992484Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9992852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9993036Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9993276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:04.9993615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:04.9994010Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9994400Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:04.9994625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:04.9994848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:04.9994932Z ok (6.252s) 2023-01-11T22:14:04.9994952Z 2023-01-11T22:14:04.9995211Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9995320Z Ran 1 test in 6.253s 2023-01-11T22:14:04.9995339Z 2023-01-11T22:14:04.9995429Z OK 2023-01-11T22:14:04.9995448Z 2023-01-11T22:14:04.9995574Z Generating XML reports... 2023-01-11T22:14:04.9996017Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220824.xml 2023-01-11T22:14:04.9996383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9996556Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9996967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9997161Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9997179Z 2023-01-11T22:14:04.9997284Z Running tests... 2023-01-11T22:14:04.9997544Z ---------------------------------------------------------------------- 2023-01-11T22:14:04.9997849Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:04.9998111Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:04.9998330Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47956 2023-01-11T22:14:04.9998542Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47957 2023-01-11T22:14:04.9998909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:04.9999067Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:04.9999439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:04.9999623Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:04.9999985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0000153Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0000523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0000705Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0000948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:05.0001173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:05.0001570Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0001959Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0002180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:05.0002402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:05.0002553Z ok (6.851s) 2023-01-11T22:14:05.0002572Z 2023-01-11T22:14:05.0002832Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0002942Z Ran 1 test in 6.852s 2023-01-11T22:14:05.0002961Z 2023-01-11T22:14:05.0003050Z OK 2023-01-11T22:14:05.0003068Z 2023-01-11T22:14:05.0003173Z Generating XML reports... 2023-01-11T22:14:05.0003621Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220832.xml 2023-01-11T22:14:05.0003987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0004158Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0004535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0004721Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0004744Z 2023-01-11T22:14:05.0004848Z Running tests... 2023-01-11T22:14:05.0005107Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0005399Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0005679Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:05.0005939Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48068 2023-01-11T22:14:05.0006156Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48069 2023-01-11T22:14:05.0006525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0006697Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0007072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0007261Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0007616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0007768Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0008138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0008320Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0008558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:05.0008796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:05.0009189Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0009580Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0009804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:05.0010026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:05.0010111Z ok (6.955s) 2023-01-11T22:14:05.0010130Z 2023-01-11T22:14:05.0010637Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0010757Z Ran 1 test in 6.956s 2023-01-11T22:14:05.0010778Z 2023-01-11T22:14:05.0010868Z OK 2023-01-11T22:14:05.0010887Z 2023-01-11T22:14:05.0011009Z Generating XML reports... 2023-01-11T22:14:05.0011459Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220842.xml 2023-01-11T22:14:05.0011825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0012094Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0012454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0012640Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0012659Z 2023-01-11T22:14:05.0012765Z Running tests... 2023-01-11T22:14:05.0013020Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0013330Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0013602Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:05.0014340Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.612s) 2023-01-11T22:14:05.0014363Z 2023-01-11T22:14:05.0014617Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0014726Z Ran 1 test in 1.613s 2023-01-11T22:14:05.0014745Z 2023-01-11T22:14:05.0014848Z OK (skipped=1) 2023-01-11T22:14:05.0014867Z 2023-01-11T22:14:05.0014972Z Generating XML reports... 2023-01-11T22:14:05.0015475Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220851.xml 2023-01-11T22:14:05.0015856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0016026Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0016398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0016586Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0016606Z 2023-01-11T22:14:05.0016711Z Running tests... 2023-01-11T22:14:05.0016968Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0017258Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0017548Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:05.0018289Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.657s) 2023-01-11T22:14:05.0018309Z 2023-01-11T22:14:05.0018563Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0018674Z Ran 1 test in 1.657s 2023-01-11T22:14:05.0018694Z 2023-01-11T22:14:05.0018798Z OK (skipped=1) 2023-01-11T22:14:05.0018816Z 2023-01-11T22:14:05.0018935Z Generating XML reports... 2023-01-11T22:14:05.0019378Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220856.xml 2023-01-11T22:14:05.0019744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0019918Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0020274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0020462Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0020481Z 2023-01-11T22:14:05.0020588Z Running tests... 2023-01-11T22:14:05.0020843Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0021212Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0021516Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:05.0021731Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48248 2023-01-11T22:14:05.0021938Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48249 2023-01-11T22:14:05.0022304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0022459Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0022828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0023015Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0023377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0023544Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0023906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0024089Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0024446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:05.0024677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:05.0025075Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0025467Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0025694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:05.0025917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:05.0026064Z skip: Need at least 4 CUDA devices (4.185s) 2023-01-11T22:14:05.0026083Z 2023-01-11T22:14:05.0026344Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0026453Z Ran 1 test in 4.185s 2023-01-11T22:14:05.0026476Z 2023-01-11T22:14:05.0026581Z OK (skipped=1) 2023-01-11T22:14:05.0026599Z 2023-01-11T22:14:05.0026704Z Generating XML reports... 2023-01-11T22:14:05.0027150Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220900.xml 2023-01-11T22:14:05.0027516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0027686Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0028057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0028245Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0028264Z 2023-01-11T22:14:05.0028369Z Running tests... 2023-01-11T22:14:05.0028633Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0028946Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0029248Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:05.0029463Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48357 2023-01-11T22:14:05.0029677Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48358 2023-01-11T22:14:05.0030042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0030268Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0030641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0030825Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0031185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0031340Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0031708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0031890Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0032130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:05.0032373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:05.0032766Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0033155Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0033438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:05.0033667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:05.0033798Z skip: Need at least 4 CUDA devices (4.147s) 2023-01-11T22:14:05.0033831Z 2023-01-11T22:14:05.0034075Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0034183Z Ran 1 test in 4.147s 2023-01-11T22:14:05.0034202Z 2023-01-11T22:14:05.0034314Z OK (skipped=1) 2023-01-11T22:14:05.0034332Z 2023-01-11T22:14:05.0034452Z Generating XML reports... 2023-01-11T22:14:05.0034898Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220906.xml 2023-01-11T22:14:05.0035263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0035434Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0035812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0035984Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0036003Z 2023-01-11T22:14:05.0036109Z Running tests... 2023-01-11T22:14:05.0036368Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0036677Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0036962Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:05.0037701Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84886 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.649s) 2023-01-11T22:14:05.0037724Z 2023-01-11T22:14:05.0037984Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0038090Z Ran 1 test in 1.649s 2023-01-11T22:14:05.0038109Z 2023-01-11T22:14:05.0038214Z OK (skipped=1) 2023-01-11T22:14:05.0038232Z 2023-01-11T22:14:05.0038352Z Generating XML reports... 2023-01-11T22:14:05.0038781Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220913.xml 2023-01-11T22:14:05.0039200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0039371Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0039745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0039932Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0039951Z 2023-01-11T22:14:05.0040059Z Running tests... 2023-01-11T22:14:05.0040319Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0040627Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0040868Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:05.0041080Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48500 2023-01-11T22:14:05.0041295Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48501 2023-01-11T22:14:05.0041663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0041835Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0042208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0042443Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0042813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0042982Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0043331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0043516Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0043760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:05.0043998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:05.0044391Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0044826Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0045052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:05.0045274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:05.0045491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T22:14:05.0045725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T22:14:05.0046119Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:05.0046505Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:05.0046833Z STAGE:2023-01-11 22:09:21 48501:48501 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0047154Z STAGE:2023-01-11 22:09:21 48500:48500 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0047484Z STAGE:2023-01-11 22:09:21 48501:48501 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0047807Z STAGE:2023-01-11 22:09:21 48500:48500 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0048151Z STAGE:2023-01-11 22:09:21 48501:48501 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0048552Z STAGE:2023-01-11 22:09:21 48500:48500 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0048858Z STAGE:2023-01-11 22:09:21 48501:48501 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0049173Z STAGE:2023-01-11 22:09:21 48500:48500 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0049712Z STAGE:2023-01-11 22:09:21 48501:48501 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 22:09:21 48500:48500 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0049733Z 2023-01-11T22:14:05.0050073Z STAGE:2023-01-11 22:09:21 48501:48501 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0050584Z STAGE:2023-01-11 22:09:21 48500:48500 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0050698Z ok (4.224s) 2023-01-11T22:14:05.0050713Z 2023-01-11T22:14:05.0050987Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0051097Z Ran 1 test in 4.224s 2023-01-11T22:14:05.0051116Z 2023-01-11T22:14:05.0051205Z OK 2023-01-11T22:14:05.0051224Z 2023-01-11T22:14:05.0051330Z Generating XML reports... 2023-01-11T22:14:05.0051778Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220917.xml 2023-01-11T22:14:05.0052219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0052399Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0052779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0052968Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0052987Z 2023-01-11T22:14:05.0053140Z Running tests... 2023-01-11T22:14:05.0053406Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0053716Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0053960Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:05.0054175Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48619 2023-01-11T22:14:05.0054391Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48620 2023-01-11T22:14:05.0054753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0054922Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0055295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0055481Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0055843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0055995Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0056360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0056543Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0056787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:05.0057029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:05.0057423Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0057812Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0058117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:05.0058340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:05.0058558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T22:14:05.0058798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T22:14:05.0059195Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:05.0059580Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:05.0059910Z STAGE:2023-01-11 22:09:28 48619:48619 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0060230Z STAGE:2023-01-11 22:09:28 48620:48620 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0060777Z STAGE:2023-01-11 22:09:28 48619:48619 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 22:09:28 48620:48620 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0060797Z 2023-01-11T22:14:05.0061403Z STAGE:2023-01-11 22:09:28 48619:48619 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 22:09:28 48620:48620 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0061425Z 2023-01-11T22:14:05.0061753Z STAGE:2023-01-11 22:09:28 48620:48620 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0062070Z STAGE:2023-01-11 22:09:28 48619:48619 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0062395Z STAGE:2023-01-11 22:09:28 48619:48619 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0062703Z STAGE:2023-01-11 22:09:28 48620:48620 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0063041Z STAGE:2023-01-11 22:09:28 48619:48619 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0063381Z STAGE:2023-01-11 22:09:28 48620:48620 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0063482Z ok (4.341s) 2023-01-11T22:14:05.0063501Z 2023-01-11T22:14:05.0063765Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0063875Z Ran 1 test in 4.341s 2023-01-11T22:14:05.0063895Z 2023-01-11T22:14:05.0063985Z OK 2023-01-11T22:14:05.0064004Z 2023-01-11T22:14:05.0064125Z Generating XML reports... 2023-01-11T22:14:05.0064553Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220924.xml 2023-01-11T22:14:05.0064919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0065094Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0065468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0065655Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0065674Z 2023-01-11T22:14:05.0065779Z Running tests... 2023-01-11T22:14:05.0066040Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0066350Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0066614Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:05.0066813Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48738 2023-01-11T22:14:05.0067025Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48739 2023-01-11T22:14:05.0067447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0067615Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0067990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0068175Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0068532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0068700Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0069050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0069238Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0069476Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:05.0069721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:05.0070117Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0070505Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0070774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:05.0071002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:05.0071237Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T22:14:05.0071455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T22:14:05.0071854Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:05.0072238Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:05.0072567Z STAGE:2023-01-11 22:09:35 48739:48739 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0072891Z STAGE:2023-01-11 22:09:35 48738:48738 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0073428Z STAGE:2023-01-11 22:09:35 48738:48738 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 22:09:35 48739:48739 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0073448Z 2023-01-11T22:14:05.0073793Z STAGE:2023-01-11 22:09:35 48738:48738 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0074132Z STAGE:2023-01-11 22:09:35 48739:48739 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0074459Z STAGE:2023-01-11 22:09:35 48739:48739 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0074773Z STAGE:2023-01-11 22:09:35 48738:48738 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0075083Z STAGE:2023-01-11 22:09:35 48738:48738 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0075633Z STAGE:2023-01-11 22:09:35 48739:48739 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 22:09:35 48738:48738 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0075653Z 2023-01-11T22:14:05.0075992Z STAGE:2023-01-11 22:09:35 48739:48739 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0076093Z ok (4.264s) 2023-01-11T22:14:05.0076113Z 2023-01-11T22:14:05.0076375Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0076552Z Ran 1 test in 4.264s 2023-01-11T22:14:05.0076572Z 2023-01-11T22:14:05.0076663Z OK 2023-01-11T22:14:05.0076682Z 2023-01-11T22:14:05.0076804Z Generating XML reports... 2023-01-11T22:14:05.0077253Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220931.xml 2023-01-11T22:14:05.0077604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0077783Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0078159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0078347Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0078366Z 2023-01-11T22:14:05.0078472Z Running tests... 2023-01-11T22:14:05.0078731Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0079043Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0079302Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:05.0079516Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48857 2023-01-11T22:14:05.0079712Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48858 2023-01-11T22:14:05.0080126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0080304Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0080684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0080869Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0081221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0081391Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0081757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0081924Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0082165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:05.0082403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:05.0082797Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0083184Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0083408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:05.0083633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:05.0083864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T22:14:05.0084096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T22:14:05.0084474Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:05.0084864Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:05.0085194Z STAGE:2023-01-11 22:09:42 48857:48857 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0085510Z STAGE:2023-01-11 22:09:42 48858:48858 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0086107Z STAGE:2023-01-11 22:09:42 48857:48857 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 22:09:42 48858:48858 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0086127Z 2023-01-11T22:14:05.0086687Z STAGE:2023-01-11 22:09:42 48858:48858 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 22:09:42 48857:48857 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0086712Z 2023-01-11T22:14:05.0087035Z STAGE:2023-01-11 22:09:42 48858:48858 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0087350Z STAGE:2023-01-11 22:09:42 48857:48857 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0087675Z STAGE:2023-01-11 22:09:42 48857:48857 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0087994Z STAGE:2023-01-11 22:09:42 48858:48858 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0088337Z STAGE:2023-01-11 22:09:42 48857:48857 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0088662Z STAGE:2023-01-11 22:09:42 48858:48858 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0088764Z ok (4.143s) 2023-01-11T22:14:05.0088783Z 2023-01-11T22:14:05.0089044Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0089153Z Ran 1 test in 4.143s 2023-01-11T22:14:05.0089218Z 2023-01-11T22:14:05.0089311Z OK 2023-01-11T22:14:05.0089330Z 2023-01-11T22:14:05.0089452Z Generating XML reports... 2023-01-11T22:14:05.0089899Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220938.xml 2023-01-11T22:14:05.0090267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0090665Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0091039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0091224Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0091244Z 2023-01-11T22:14:05.0091349Z Running tests... 2023-01-11T22:14:05.0091609Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0091923Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0092175Z test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:05.0092387Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48976 2023-01-11T22:14:05.0092595Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48977 2023-01-11T22:14:05.0092947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0093118Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0093493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0093678Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0094039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0094208Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0094583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0094762Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0094987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:05.0095309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:05.0095705Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0096095Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0096318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:05.0096542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:05.0096699Z skip: Skipped due to small world size. (4.252s) 2023-01-11T22:14:05.0096719Z 2023-01-11T22:14:05.0096980Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0097089Z Ran 1 test in 4.252s 2023-01-11T22:14:05.0097108Z 2023-01-11T22:14:05.0097196Z OK (skipped=1) 2023-01-11T22:14:05.0097230Z 2023-01-11T22:14:05.0097336Z Generating XML reports... 2023-01-11T22:14:05.0097778Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220945.xml 2023-01-11T22:14:05.0098143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0098315Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0098753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0098950Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0098969Z 2023-01-11T22:14:05.0099074Z Running tests... 2023-01-11T22:14:05.0099339Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0099628Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0099885Z test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:05.0100102Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49085 2023-01-11T22:14:05.0100313Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49086 2023-01-11T22:14:05.0100678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0100852Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0101226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0101412Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0101754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0101922Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0102291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0102474Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0102714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:05.0102956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:05.0103347Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0103736Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0103957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:05.0104163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:05.0104372Z skip: Skipped due to small world size. (4.290s) 2023-01-11T22:14:05.0104392Z 2023-01-11T22:14:05.0104655Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0104766Z Ran 1 test in 4.290s 2023-01-11T22:14:05.0104785Z 2023-01-11T22:14:05.0104889Z OK (skipped=1) 2023-01-11T22:14:05.0104907Z 2023-01-11T22:14:05.0105028Z Generating XML reports... 2023-01-11T22:14:05.0105479Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220951.xml 2023-01-11T22:14:05.0105844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0106016Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0106372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0106563Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0106582Z 2023-01-11T22:14:05.0106689Z Running tests... 2023-01-11T22:14:05.0106947Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0107255Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0107515Z test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:05.0107777Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49194 2023-01-11T22:14:05.0107997Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49195 2023-01-11T22:14:05.0108346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0108519Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0108892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0109078Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0109435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0109604Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0109974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0110154Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0110393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:05.0110615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:05.0111009Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0111405Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0111630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:05.0111851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:05.0112008Z skip: Skipped due to small world size. (4.271s) 2023-01-11T22:14:05.0112028Z 2023-01-11T22:14:05.0112289Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0112399Z Ran 1 test in 4.271s 2023-01-11T22:14:05.0112418Z 2023-01-11T22:14:05.0112522Z OK (skipped=1) 2023-01-11T22:14:05.0112542Z 2023-01-11T22:14:05.0112647Z Generating XML reports... 2023-01-11T22:14:05.0113090Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220958.xml 2023-01-11T22:14:05.0113516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0113687Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0114061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0114249Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0114272Z 2023-01-11T22:14:05.0114378Z Running tests... 2023-01-11T22:14:05.0114636Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0114929Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0115179Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:05.0115389Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49303 2023-01-11T22:14:05.0115602Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49304 2023-01-11T22:14:05.0115970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0116140Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0116514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0116748Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0117112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0117266Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0117632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0117821Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0118058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:05.0118294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:05.0118689Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0119082Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0119304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:05.0119512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:05.0119668Z skip: Skipped due to small world size. (4.153s) 2023-01-11T22:14:05.0119687Z 2023-01-11T22:14:05.0119954Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0120062Z Ran 1 test in 4.153s 2023-01-11T22:14:05.0120081Z 2023-01-11T22:14:05.0120184Z OK (skipped=1) 2023-01-11T22:14:05.0120203Z 2023-01-11T22:14:05.0120323Z Generating XML reports... 2023-01-11T22:14:05.0120765Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221005.xml 2023-01-11T22:14:05.0121134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0121304Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0121661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0121845Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0121864Z 2023-01-11T22:14:05.0121970Z Running tests... 2023-01-11T22:14:05.0122281Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0122591Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0122832Z test_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:05.0123043Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49412 2023-01-11T22:14:05.0123252Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49413 2023-01-11T22:14:05.0123619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0123772Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0124144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0124328Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0124691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0124859Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0125223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0125406Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0125692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:05.0125919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:05.0126316Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0126705Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0126931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:05.0127153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:05.0127679Z STAGE:2023-01-11 22:10:16 49413:49413 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 22:10:16 49412:49412 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0127699Z 2023-01-11T22:14:05.0128237Z STAGE:2023-01-11 22:10:16 49413:49413 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 22:10:16 49412:49412 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0128257Z 2023-01-11T22:14:05.0128822Z STAGE:2023-01-11 22:10:16 49412:49412 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 22:10:16 49413:49413 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0128845Z 2023-01-11T22:14:05.0129167Z STAGE:2023-01-11 22:10:16 49413:49413 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0129486Z STAGE:2023-01-11 22:10:16 49412:49412 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0130017Z STAGE:2023-01-11 22:10:16 49412:49412 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 22:10:16 49413:49413 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0130039Z 2023-01-11T22:14:05.0130603Z STAGE:2023-01-11 22:10:16 49413:49413 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0130942Z STAGE:2023-01-11 22:10:16 49412:49412 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0131041Z ok (4.228s) 2023-01-11T22:14:05.0131062Z 2023-01-11T22:14:05.0131322Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0131513Z Ran 1 test in 4.228s 2023-01-11T22:14:05.0131532Z 2023-01-11T22:14:05.0131622Z OK 2023-01-11T22:14:05.0131641Z 2023-01-11T22:14:05.0131764Z Generating XML reports... 2023-01-11T22:14:05.0132213Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221012.xml 2023-01-11T22:14:05.0132577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0132753Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0133114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0133302Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0133321Z 2023-01-11T22:14:05.0133427Z Running tests... 2023-01-11T22:14:05.0133688Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0134002Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0134244Z test_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:05.0134456Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49525 2023-01-11T22:14:05.0134666Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49526 2023-01-11T22:14:05.0135125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0135307Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0135683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0135869Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0136225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0136397Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0136764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0136947Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0137189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:05.0137418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:05.0137813Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0138205Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0138431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:05.0138658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:05.0138986Z STAGE:2023-01-11 22:10:22 49525:49525 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0139303Z STAGE:2023-01-11 22:10:22 49526:49526 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0139639Z STAGE:2023-01-11 22:10:22 49526:49526 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0139970Z STAGE:2023-01-11 22:10:22 49525:49525 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0140296Z STAGE:2023-01-11 22:10:22 49526:49526 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0140638Z STAGE:2023-01-11 22:10:22 49525:49525 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0140958Z STAGE:2023-01-11 22:10:22 49526:49526 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0141346Z STAGE:2023-01-11 22:10:22 49525:49525 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0141674Z STAGE:2023-01-11 22:10:22 49525:49525 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0141998Z STAGE:2023-01-11 22:10:22 49526:49526 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0142340Z STAGE:2023-01-11 22:10:22 49525:49525 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0142679Z STAGE:2023-01-11 22:10:22 49526:49526 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0142763Z ok (4.249s) 2023-01-11T22:14:05.0142797Z 2023-01-11T22:14:05.0143039Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0143150Z Ran 1 test in 4.249s 2023-01-11T22:14:05.0143169Z 2023-01-11T22:14:05.0143260Z OK 2023-01-11T22:14:05.0143283Z 2023-01-11T22:14:05.0143405Z Generating XML reports... 2023-01-11T22:14:05.0143855Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221019.xml 2023-01-11T22:14:05.0144221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0144394Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0144871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0145048Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0145084Z 2023-01-11T22:14:05.0145175Z Running tests... 2023-01-11T22:14:05.0145434Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0145743Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0146019Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports reduce multigpu (0.002s) 2023-01-11T22:14:05.0146038Z 2023-01-11T22:14:05.0146294Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0146403Z Ran 1 test in 0.002s 2023-01-11T22:14:05.0146423Z 2023-01-11T22:14:05.0146526Z OK (skipped=1) 2023-01-11T22:14:05.0146545Z 2023-01-11T22:14:05.0146664Z Generating XML reports... 2023-01-11T22:14:05.0147094Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221025.xml 2023-01-11T22:14:05.0147457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0147629Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0148002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0148187Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0148206Z 2023-01-11T22:14:05.0148312Z Running tests... 2023-01-11T22:14:05.0148569Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0148876Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0149110Z test_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:05.0149327Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49671 2023-01-11T22:14:05.0149538Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49672 2023-01-11T22:14:05.0149906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0150075Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0150448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0150688Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0151049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0151216Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0151569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0151754Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0151992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:05.0152227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:05.0152621Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0153015Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0153241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:05.0153465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:05.0153839Z STAGE:2023-01-11 22:10:32 49672:49672 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0154151Z STAGE:2023-01-11 22:10:32 49671:49671 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0154478Z STAGE:2023-01-11 22:10:32 49672:49672 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0154805Z STAGE:2023-01-11 22:10:32 49671:49671 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0155144Z STAGE:2023-01-11 22:10:32 49672:49672 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0155490Z STAGE:2023-01-11 22:10:32 49671:49671 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0155810Z STAGE:2023-01-11 22:10:32 49672:49672 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0156125Z STAGE:2023-01-11 22:10:32 49671:49671 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0156660Z STAGE:2023-01-11 22:10:32 49671:49671 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 22:10:32 49672:49672 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0156681Z 2023-01-11T22:14:05.0157239Z STAGE:2023-01-11 22:10:32 49671:49671 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 22:10:32 49672:49672 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0157258Z 2023-01-11T22:14:05.0157361Z ok (4.230s) 2023-01-11T22:14:05.0157381Z 2023-01-11T22:14:05.0157626Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0157738Z Ran 1 test in 4.230s 2023-01-11T22:14:05.0157757Z 2023-01-11T22:14:05.0157849Z OK 2023-01-11T22:14:05.0157868Z 2023-01-11T22:14:05.0157987Z Generating XML reports... 2023-01-11T22:14:05.0158437Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221028.xml 2023-01-11T22:14:05.0158805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0158979Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0159357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0159544Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0159613Z 2023-01-11T22:14:05.0159708Z Running tests... 2023-01-11T22:14:05.0159970Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0160279Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0160568Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce_scatter_tensor (0.002s) 2023-01-11T22:14:05.0160587Z 2023-01-11T22:14:05.0160849Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0160956Z Ran 1 test in 0.002s 2023-01-11T22:14:05.0160975Z 2023-01-11T22:14:05.0161080Z OK (skipped=1) 2023-01-11T22:14:05.0161099Z 2023-01-11T22:14:05.0161220Z Generating XML reports... 2023-01-11T22:14:05.0161648Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221035.xml 2023-01-11T22:14:05.0162016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0162190Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0162567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0162753Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0162772Z 2023-01-11T22:14:05.0162878Z Running tests... 2023-01-11T22:14:05.0163184Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0163502Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0163772Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports reduce_scatter_v (0.003s) 2023-01-11T22:14:05.0163792Z 2023-01-11T22:14:05.0164031Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0164139Z Ran 1 test in 0.003s 2023-01-11T22:14:05.0164162Z 2023-01-11T22:14:05.0164268Z OK (skipped=1) 2023-01-11T22:14:05.0164287Z 2023-01-11T22:14:05.0164407Z Generating XML reports... 2023-01-11T22:14:05.0164851Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221037.xml 2023-01-11T22:14:05.0165216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0165393Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0165768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0165955Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0165974Z 2023-01-11T22:14:05.0166065Z Running tests... 2023-01-11T22:14:05.0166327Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0166638Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0166885Z test_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:05.0167101Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49850 2023-01-11T22:14:05.0167314Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49851 2023-01-11T22:14:05.0167681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0188123Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0188623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0188803Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0189166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0189494Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0189871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0190050Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0190284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:05.0190522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:05.0190911Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0191293Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0191505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:05.0191723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:05.0192048Z STAGE:2023-01-11 22:10:43 49851:49851 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0192358Z STAGE:2023-01-11 22:10:43 49850:49850 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0192741Z STAGE:2023-01-11 22:10:43 49851:49851 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0193067Z STAGE:2023-01-11 22:10:43 49850:49850 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0193399Z STAGE:2023-01-11 22:10:43 49851:49851 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0193731Z STAGE:2023-01-11 22:10:43 49850:49850 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0194041Z STAGE:2023-01-11 22:10:43 49851:49851 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0194343Z STAGE:2023-01-11 22:10:43 49850:49850 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0194657Z STAGE:2023-01-11 22:10:43 49851:49851 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0194966Z STAGE:2023-01-11 22:10:43 49850:49850 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0195292Z STAGE:2023-01-11 22:10:43 49851:49851 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0195618Z STAGE:2023-01-11 22:10:43 49850:49850 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0195709Z ok (4.265s) 2023-01-11T22:14:05.0195734Z 2023-01-11T22:14:05.0195990Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0196092Z Ran 1 test in 4.265s 2023-01-11T22:14:05.0196112Z 2023-01-11T22:14:05.0196186Z OK 2023-01-11T22:14:05.0196207Z 2023-01-11T22:14:05.0196321Z Generating XML reports... 2023-01-11T22:14:05.0196766Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221039.xml 2023-01-11T22:14:05.0197122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0197285Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0197653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0197831Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0197851Z 2023-01-11T22:14:05.0197948Z Running tests... 2023-01-11T22:14:05.0198201Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0198494Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0198807Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s) 2023-01-11T22:14:05.0198827Z 2023-01-11T22:14:05.0199074Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0199174Z Ran 1 test in 0.002s 2023-01-11T22:14:05.0199194Z 2023-01-11T22:14:05.0199290Z OK (skipped=1) 2023-01-11T22:14:05.0199309Z 2023-01-11T22:14:05.0199420Z Generating XML reports... 2023-01-11T22:14:05.0199863Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221046.xml 2023-01-11T22:14:05.0200219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0200380Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0200739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0200919Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0200938Z 2023-01-11T22:14:05.0201035Z Running tests... 2023-01-11T22:14:05.0201285Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0201584Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0201835Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s) 2023-01-11T22:14:05.0201904Z 2023-01-11T22:14:05.0202160Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0202262Z Ran 1 test in 0.002s 2023-01-11T22:14:05.0202282Z 2023-01-11T22:14:05.0202373Z OK (skipped=1) 2023-01-11T22:14:05.0202399Z 2023-01-11T22:14:05.0202505Z Generating XML reports... 2023-01-11T22:14:05.0202941Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221049.xml 2023-01-11T22:14:05.0203301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0203463Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0203827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0204003Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0204022Z 2023-01-11T22:14:05.0204122Z Running tests... 2023-01-11T22:14:05.0204372Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0204664Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0204905Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:05.0205110Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50029 2023-01-11T22:14:05.0205318Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50030 2023-01-11T22:14:05.0205676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0205837Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0206200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0206378Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0206727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0206890Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0207251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0207427Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0207715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:05.0207944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:05.0208333Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0208717Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0208932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:05.0209137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:05.0209460Z STAGE:2023-01-11 22:10:55 50029:50029 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0209772Z STAGE:2023-01-11 22:10:55 50030:50030 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0210098Z STAGE:2023-01-11 22:10:55 50030:50030 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0210635Z STAGE:2023-01-11 22:10:55 50029:50029 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0211278Z STAGE:2023-01-11 22:10:55 50030:50030 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 22:10:55 50029:50029 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0211303Z 2023-01-11T22:14:05.0211622Z STAGE:2023-01-11 22:10:55 50030:50030 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0211926Z STAGE:2023-01-11 22:10:55 50029:50029 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0212238Z STAGE:2023-01-11 22:10:55 50029:50029 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0212557Z STAGE:2023-01-11 22:10:55 50030:50030 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0212882Z STAGE:2023-01-11 22:10:55 50029:50029 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0213211Z STAGE:2023-01-11 22:10:55 50030:50030 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0213303Z ok (4.249s) 2023-01-11T22:14:05.0213323Z 2023-01-11T22:14:05.0213578Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0213679Z Ran 1 test in 4.249s 2023-01-11T22:14:05.0213699Z 2023-01-11T22:14:05.0213788Z OK 2023-01-11T22:14:05.0213807Z 2023-01-11T22:14:05.0213925Z Generating XML reports... 2023-01-11T22:14:05.0214377Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221051.xml 2023-01-11T22:14:05.0214725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0214893Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0215259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0215436Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0215455Z 2023-01-11T22:14:05.0215564Z Running tests... 2023-01-11T22:14:05.0215823Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0216136Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0216376Z test_scatter (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:05.0216585Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50142 2023-01-11T22:14:05.0216783Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50143 2023-01-11T22:14:05.0217218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0217382Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0217747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0217925Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0218284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0218445Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0218809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0218979Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0219211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:05.0219445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:05.0219830Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0220212Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0220476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:05.0220698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:05.0221023Z STAGE:2023-01-11 22:11:02 50142:50142 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0221333Z STAGE:2023-01-11 22:11:02 50143:50143 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0221651Z STAGE:2023-01-11 22:11:02 50142:50142 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0221965Z STAGE:2023-01-11 22:11:02 50143:50143 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0222296Z STAGE:2023-01-11 22:11:02 50142:50142 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0222625Z STAGE:2023-01-11 22:11:02 50143:50143 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0222936Z STAGE:2023-01-11 22:11:02 50143:50143 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0223248Z STAGE:2023-01-11 22:11:02 50142:50142 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0223779Z STAGE:2023-01-11 22:11:02 50142:50142 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 22:11:02 50143:50143 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0223800Z 2023-01-11T22:14:05.0224142Z STAGE:2023-01-11 22:11:02 50142:50142 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0224480Z STAGE:2023-01-11 22:11:02 50143:50143 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0224583Z ok (4.260s) 2023-01-11T22:14:05.0224603Z 2023-01-11T22:14:05.0224848Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0224960Z Ran 1 test in 4.260s 2023-01-11T22:14:05.0224980Z 2023-01-11T22:14:05.0225075Z OK 2023-01-11T22:14:05.0225095Z 2023-01-11T22:14:05.0225216Z Generating XML reports... 2023-01-11T22:14:05.0225665Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221058.xml 2023-01-11T22:14:05.0226034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0226205Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0226663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0226833Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0226870Z 2023-01-11T22:14:05.0226961Z Running tests... 2023-01-11T22:14:05.0227222Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0227537Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0227788Z test_scatter_checks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:05.0228003Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50255 2023-01-11T22:14:05.0228220Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50256 2023-01-11T22:14:05.0228587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0228760Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0229117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0229305Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0229668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0229902Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0230283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0230470Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0230710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:05.0230950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:05.0231330Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0231721Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0231949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:05.0232175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:05.0232273Z ok (4.247s) 2023-01-11T22:14:05.0232293Z 2023-01-11T22:14:05.0232561Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0232675Z Ran 1 test in 4.248s 2023-01-11T22:14:05.0232694Z 2023-01-11T22:14:05.0232786Z OK 2023-01-11T22:14:05.0232805Z 2023-01-11T22:14:05.0232926Z Generating XML reports... 2023-01-11T22:14:05.0233358Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221105.xml 2023-01-11T22:14:05.0233727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0233900Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0234277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0234466Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0234486Z 2023-01-11T22:14:05.0234593Z Running tests... 2023-01-11T22:14:05.0234851Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0235162Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0235414Z test_scatter_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:05.0235671Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50364 2023-01-11T22:14:05.0235883Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50365 2023-01-11T22:14:05.0236252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0236425Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0236797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0236982Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0237344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0237519Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0237875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0238066Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0238308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:05.0238549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:05.0239044Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0239449Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0239677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:05.0239904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:05.0240241Z STAGE:2023-01-11 22:11:15 50364:50364 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0240541Z STAGE:2023-01-11 22:11:15 50365:50365 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0240870Z STAGE:2023-01-11 22:11:15 50365:50365 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0241193Z STAGE:2023-01-11 22:11:15 50364:50364 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0241538Z STAGE:2023-01-11 22:11:15 50365:50365 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0241880Z STAGE:2023-01-11 22:11:15 50364:50364 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0242200Z STAGE:2023-01-11 22:11:15 50365:50365 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0242514Z STAGE:2023-01-11 22:11:15 50364:50364 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0242845Z STAGE:2023-01-11 22:11:15 50365:50365 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0243392Z STAGE:2023-01-11 22:11:15 50364:50364 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 22:11:15 50365:50365 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0243414Z 2023-01-11T22:14:05.0243758Z STAGE:2023-01-11 22:11:15 50364:50364 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0243844Z ok (4.156s) 2023-01-11T22:14:05.0243863Z 2023-01-11T22:14:05.0244130Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0244241Z Ran 1 test in 4.156s 2023-01-11T22:14:05.0244260Z 2023-01-11T22:14:05.0244353Z OK 2023-01-11T22:14:05.0244372Z 2023-01-11T22:14:05.0244496Z Generating XML reports... 2023-01-11T22:14:05.0244993Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221112.xml 2023-01-11T22:14:05.0245426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0245601Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0245959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0246147Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0246172Z 2023-01-11T22:14:05.0246280Z Running tests... 2023-01-11T22:14:05.0246544Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0246853Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0247105Z test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s) 2023-01-11T22:14:05.0247125Z 2023-01-11T22:14:05.0247383Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0247501Z Ran 1 test in 0.002s 2023-01-11T22:14:05.0247521Z 2023-01-11T22:14:05.0247629Z OK (skipped=1) 2023-01-11T22:14:05.0247648Z 2023-01-11T22:14:05.0247754Z Generating XML reports... 2023-01-11T22:14:05.0248196Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221118.xml 2023-01-11T22:14:05.0248602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0248778Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0249157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0249342Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0249362Z 2023-01-11T22:14:05.0249467Z Running tests... 2023-01-11T22:14:05.0249728Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0250026Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0250451Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s) 2023-01-11T22:14:05.0250474Z 2023-01-11T22:14:05.0250747Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0250859Z Ran 1 test in 0.002s 2023-01-11T22:14:05.0250884Z 2023-01-11T22:14:05.0250990Z OK (skipped=1) 2023-01-11T22:14:05.0251009Z 2023-01-11T22:14:05.0251132Z Generating XML reports... 2023-01-11T22:14:05.0251578Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221121.xml 2023-01-11T22:14:05.0251946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0252119Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0252479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0252665Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0252684Z 2023-01-11T22:14:05.0252791Z Running tests... 2023-01-11T22:14:05.0253045Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0253357Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0253612Z test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:05.0253831Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50543 2023-01-11T22:14:05.0254041Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50544 2023-01-11T22:14:05.0254405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0254657Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0255034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0255219Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0255582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0255755Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0256131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0256318Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0256556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:05.0256785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:05.0257187Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0257578Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0257865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:05.0258094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:05.0258333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T22:14:05.0258570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T22:14:05.0258964Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:05.0259354Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:05.0259670Z STAGE:2023-01-11 22:11:27 50544:50544 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0259990Z STAGE:2023-01-11 22:11:27 50543:50543 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0260326Z STAGE:2023-01-11 22:11:27 50543:50543 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0260650Z STAGE:2023-01-11 22:11:27 50544:50544 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0261212Z STAGE:2023-01-11 22:11:27 50543:50543 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 22:11:27 50544:50544 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0261233Z 2023-01-11T22:14:05.0261757Z STAGE:2023-01-11 22:11:27 50544:50544 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 22:11:27 50543:50543 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0261777Z 2023-01-11T22:14:05.0262106Z STAGE:2023-01-11 22:11:27 50544:50544 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0262651Z STAGE:2023-01-11 22:11:27 50544:50544 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 22:11:27 50543:50543 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0262671Z 2023-01-11T22:14:05.0263016Z STAGE:2023-01-11 22:11:27 50543:50543 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0263118Z ok (4.271s) 2023-01-11T22:14:05.0263138Z 2023-01-11T22:14:05.0263400Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0263495Z Ran 1 test in 4.271s 2023-01-11T22:14:05.0263532Z 2023-01-11T22:14:05.0263661Z OK 2023-01-11T22:14:05.0263680Z 2023-01-11T22:14:05.0263802Z Generating XML reports... 2023-01-11T22:14:05.0264250Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221123.xml 2023-01-11T22:14:05.0264617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0264788Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0265166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0265353Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0265372Z 2023-01-11T22:14:05.0265480Z Running tests... 2023-01-11T22:14:05.0265727Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0266038Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0266290Z test_scatter_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:05.0266505Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50662 2023-01-11T22:14:05.0266718Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50663 2023-01-11T22:14:05.0267086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0267307Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0267695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0267884Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0268233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0268410Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0268778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0268960Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0269201Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:05.0269438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:05.0269831Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0270225Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0270433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:05.0270661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:05.0270819Z skip: Skipped due to small world size. (4.250s) 2023-01-11T22:14:05.0270839Z 2023-01-11T22:14:05.0271106Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0271219Z Ran 1 test in 4.250s 2023-01-11T22:14:05.0271239Z 2023-01-11T22:14:05.0271346Z OK (skipped=1) 2023-01-11T22:14:05.0271366Z 2023-01-11T22:14:05.0271487Z Generating XML reports... 2023-01-11T22:14:05.0271931Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221130.xml 2023-01-11T22:14:05.0272299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0272454Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0272828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0273067Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0273086Z 2023-01-11T22:14:05.0273194Z Running tests... 2023-01-11T22:14:05.0273457Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0273769Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0274030Z test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:05.0274248Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50771 2023-01-11T22:14:05.0274445Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50772 2023-01-11T22:14:05.0274814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0274985Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0275362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0275547Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0275908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0276075Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0276496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0276685Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0276910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:05.0277151Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:05.0277550Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0277939Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0278164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:05.0278388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:05.0278491Z ok (4.263s) 2023-01-11T22:14:05.0278512Z 2023-01-11T22:14:05.0278774Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0278886Z Ran 1 test in 4.263s 2023-01-11T22:14:05.0278905Z 2023-01-11T22:14:05.0278981Z OK 2023-01-11T22:14:05.0279000Z 2023-01-11T22:14:05.0279121Z Generating XML reports... 2023-01-11T22:14:05.0279571Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221137.xml 2023-01-11T22:14:05.0279941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0280112Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0280489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0280675Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0280698Z 2023-01-11T22:14:05.0280806Z Running tests... 2023-01-11T22:14:05.0281051Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0281356Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0281596Z test_send_recv (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:05.0281811Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50880 2023-01-11T22:14:05.0282080Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50881 2023-01-11T22:14:05.0282451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0282627Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0283003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0283192Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0283539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0283712Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0284082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0284275Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0284516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:05.0284755Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:05.0285155Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0285600Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0285813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:05.0286035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:05.0286137Z ok (4.228s) 2023-01-11T22:14:05.0286157Z 2023-01-11T22:14:05.0286424Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0286539Z Ran 1 test in 4.228s 2023-01-11T22:14:05.0286558Z 2023-01-11T22:14:05.0286650Z OK 2023-01-11T22:14:05.0286668Z 2023-01-11T22:14:05.0286790Z Generating XML reports... 2023-01-11T22:14:05.0287237Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221144.xml 2023-01-11T22:14:05.0287601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0287762Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0288132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0288316Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0288335Z 2023-01-11T22:14:05.0288441Z Running tests... 2023-01-11T22:14:05.0288702Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0289014Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0289273Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:05.0289489Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50989 2023-01-11T22:14:05.0289684Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50990 2023-01-11T22:14:05.0290056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0290229Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0290859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0291047Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0291408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0291664Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0292042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0292228Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0292455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:05.0292692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:05.0293088Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0293477Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0293706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:05.0293929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:05.0294027Z ok (4.224s) 2023-01-11T22:14:05.0294048Z 2023-01-11T22:14:05.0294314Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0294409Z Ran 1 test in 4.224s 2023-01-11T22:14:05.0294447Z 2023-01-11T22:14:05.0294522Z OK 2023-01-11T22:14:05.0294616Z 2023-01-11T22:14:05.0294748Z Generating XML reports... 2023-01-11T22:14:05.0295197Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221150.xml 2023-01-11T22:14:05.0295563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0295738Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0296114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0296309Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0296329Z 2023-01-11T22:14:05.0296435Z Running tests... 2023-01-11T22:14:05.0296679Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0296990Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0297277Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:05.0297493Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51098 2023-01-11T22:14:05.0297705Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51099 2023-01-11T22:14:05.0298070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0298246Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0298619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0298807Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0299156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0299331Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0299702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0299887Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0300131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:05.0300374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:05.0300825Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0301212Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0301421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:05.0301648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:05.0301982Z STAGE:2023-01-11 22:12:01 51099:51099 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0302302Z STAGE:2023-01-11 22:12:01 51098:51098 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0302633Z STAGE:2023-01-11 22:12:01 51098:51098 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0302979Z STAGE:2023-01-11 22:12:01 51098:51098 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0303308Z STAGE:2023-01-11 22:12:01 51099:51099 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0303649Z STAGE:2023-01-11 22:12:01 51099:51099 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0303749Z ok (4.246s) 2023-01-11T22:14:05.0303769Z 2023-01-11T22:14:05.0304062Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0304177Z Ran 1 test in 4.247s 2023-01-11T22:14:05.0304197Z 2023-01-11T22:14:05.0304284Z OK 2023-01-11T22:14:05.0304303Z 2023-01-11T22:14:05.0304422Z Generating XML reports... 2023-01-11T22:14:05.0304869Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221157.xml 2023-01-11T22:14:05.0305232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0305410Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0305784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0305973Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0305993Z 2023-01-11T22:14:05.0306084Z Running tests... 2023-01-11T22:14:05.0306350Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0306659Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0306935Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:05.0307149Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51211 2023-01-11T22:14:05.0307359Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51212 2023-01-11T22:14:05.0307727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0307899Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0308254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0308440Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0308804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0308975Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0309346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0309534Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0309776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:05.0310071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:05.0310467Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0310840Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0311072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:05.0311296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:05.0311630Z STAGE:2023-01-11 22:12:08 51211:51211 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0311949Z STAGE:2023-01-11 22:12:08 51212:51212 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0312285Z STAGE:2023-01-11 22:12:08 51211:51211 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0312626Z STAGE:2023-01-11 22:12:08 51211:51211 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0312958Z STAGE:2023-01-11 22:12:08 51212:51212 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0313299Z STAGE:2023-01-11 22:12:08 51212:51212 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0313430Z ok (4.280s) 2023-01-11T22:14:05.0313452Z 2023-01-11T22:14:05.0313721Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0313832Z Ran 1 test in 4.280s 2023-01-11T22:14:05.0313851Z 2023-01-11T22:14:05.0313942Z OK 2023-01-11T22:14:05.0313961Z 2023-01-11T22:14:05.0314084Z Generating XML reports... 2023-01-11T22:14:05.0314527Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221204.xml 2023-01-11T22:14:05.0314892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0315064Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0315422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0315605Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0315627Z 2023-01-11T22:14:05.0315733Z Running tests... 2023-01-11T22:14:05.0315992Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0316298Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0316563Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:05.0316778Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51324 2023-01-11T22:14:05.0316992Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51325 2023-01-11T22:14:05.0317355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0317512Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0317888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0318074Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0318436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0318607Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0318979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0319218Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0319460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:05.0319683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:05.0320082Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0320475Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0320701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:05.0320923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:05.0321256Z STAGE:2023-01-11 22:12:15 51324:51324 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0321579Z STAGE:2023-01-11 22:12:15 51325:51325 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0321911Z STAGE:2023-01-11 22:12:15 51325:51325 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0322253Z STAGE:2023-01-11 22:12:15 51325:51325 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0322610Z STAGE:2023-01-11 22:12:15 51324:51324 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0322960Z STAGE:2023-01-11 22:12:15 51324:51324 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0323061Z ok (4.269s) 2023-01-11T22:14:05.0323081Z 2023-01-11T22:14:05.0323345Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0323457Z Ran 1 test in 4.269s 2023-01-11T22:14:05.0323476Z 2023-01-11T22:14:05.0323569Z OK 2023-01-11T22:14:05.0323588Z 2023-01-11T22:14:05.0323709Z Generating XML reports... 2023-01-11T22:14:05.0324158Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221211.xml 2023-01-11T22:14:05.0324517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0324675Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0325054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0325240Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0325259Z 2023-01-11T22:14:05.0325370Z Running tests... 2023-01-11T22:14:05.0325629Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0325940Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0326170Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s) 2023-01-11T22:14:05.0326193Z 2023-01-11T22:14:05.0326450Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0326544Z Ran 1 test in 0.002s 2023-01-11T22:14:05.0326573Z 2023-01-11T22:14:05.0326664Z OK (skipped=1) 2023-01-11T22:14:05.0326683Z 2023-01-11T22:14:05.0326805Z Generating XML reports... 2023-01-11T22:14:05.0327253Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221217.xml 2023-01-11T22:14:05.0327619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0327786Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0328157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0328340Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0328411Z 2023-01-11T22:14:05.0328522Z Running tests... 2023-01-11T22:14:05.0328766Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0329078Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0329335Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s) 2023-01-11T22:14:05.0329355Z 2023-01-11T22:14:05.0329619Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0329726Z Ran 1 test in 0.002s 2023-01-11T22:14:05.0329745Z 2023-01-11T22:14:05.0329853Z OK (skipped=1) 2023-01-11T22:14:05.0329871Z 2023-01-11T22:14:05.0329994Z Generating XML reports... 2023-01-11T22:14:05.0330632Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221220.xml 2023-01-11T22:14:05.0331006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0331168Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0331543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0331731Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0331752Z 2023-01-11T22:14:05.0331860Z Running tests... 2023-01-11T22:14:05.0332194Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0332516Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0332772Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s) 2023-01-11T22:14:05.0332791Z 2023-01-11T22:14:05.0333051Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0333161Z Ran 1 test in 0.002s 2023-01-11T22:14:05.0333184Z 2023-01-11T22:14:05.0333274Z OK (skipped=1) 2023-01-11T22:14:05.0333293Z 2023-01-11T22:14:05.0333409Z Generating XML reports... 2023-01-11T22:14:05.0333849Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221222.xml 2023-01-11T22:14:05.0334212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0334382Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0334754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0334938Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0334957Z 2023-01-11T22:14:05.0335062Z Running tests... 2023-01-11T22:14:05.0335301Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0335609Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0335876Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:05.0336091Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51536 2023-01-11T22:14:05.0336302Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51537 2023-01-11T22:14:05.0336671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0336844Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0337223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0337409Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0337756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0338004Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0338375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0338557Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0338792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:05.0339036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:05.0339432Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0339816Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0340024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:05.0340249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:05.0340582Z STAGE:2023-01-11 22:12:28 51536:51536 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0340900Z STAGE:2023-01-11 22:12:28 51537:51537 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0341277Z STAGE:2023-01-11 22:12:28 51537:51537 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0341624Z STAGE:2023-01-11 22:12:28 51537:51537 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0341949Z STAGE:2023-01-11 22:12:28 51536:51536 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0342288Z STAGE:2023-01-11 22:12:28 51536:51536 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0342386Z ok (4.233s) 2023-01-11T22:14:05.0342410Z 2023-01-11T22:14:05.0342659Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0342770Z Ran 1 test in 4.233s 2023-01-11T22:14:05.0342789Z 2023-01-11T22:14:05.0342879Z OK 2023-01-11T22:14:05.0342898Z 2023-01-11T22:14:05.0343021Z Generating XML reports... 2023-01-11T22:14:05.0343469Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221225.xml 2023-01-11T22:14:05.0343839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0344012Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0344385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0344572Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0344592Z 2023-01-11T22:14:05.0344729Z Running tests... 2023-01-11T22:14:05.0344997Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0345306Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0345561Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:05.0345771Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51649 2023-01-11T22:14:05.0345988Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51650 2023-01-11T22:14:05.0346354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0346525Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0346878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0347062Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0347486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0347654Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0348026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0348216Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0348459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:05.0348699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:05.0349097Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0349471Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0349700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:05.0349918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:05.0350020Z ok (4.254s) 2023-01-11T22:14:05.0350040Z 2023-01-11T22:14:05.0350302Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0350455Z Ran 1 test in 4.254s 2023-01-11T22:14:05.0350476Z 2023-01-11T22:14:05.0350568Z OK 2023-01-11T22:14:05.0350587Z 2023-01-11T22:14:05.0350709Z Generating XML reports... 2023-01-11T22:14:05.0351139Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221231.xml 2023-01-11T22:14:05.0351503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0351673Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0352053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0352241Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0352260Z 2023-01-11T22:14:05.0352367Z Running tests... 2023-01-11T22:14:05.0352626Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0352934Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0353258Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:05.0353458Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51758 2023-01-11T22:14:05.0353670Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51759 2023-01-11T22:14:05.0354037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0354203Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0354564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0354741Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0355097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0355259Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0355616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0355791Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0356021Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:05.0356314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:05.0356710Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0357098Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0357329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:05.0357554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:05.0357887Z STAGE:2023-01-11 22:12:42 51759:51759 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0358186Z STAGE:2023-01-11 22:12:42 51758:51758 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0358518Z STAGE:2023-01-11 22:12:42 51758:51758 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0358866Z STAGE:2023-01-11 22:12:42 51758:51758 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0359194Z STAGE:2023-01-11 22:12:42 51759:51759 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0359536Z STAGE:2023-01-11 22:12:42 51759:51759 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0359635Z ok (4.267s) 2023-01-11T22:14:05.0359718Z 2023-01-11T22:14:05.0359991Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0360106Z Ran 1 test in 4.267s 2023-01-11T22:14:05.0360125Z 2023-01-11T22:14:05.0360216Z OK 2023-01-11T22:14:05.0360235Z 2023-01-11T22:14:05.0360341Z Generating XML reports... 2023-01-11T22:14:05.0360787Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221238.xml 2023-01-11T22:14:05.0361152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0361328Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0361700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0361884Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0361904Z 2023-01-11T22:14:05.0362007Z Running tests... 2023-01-11T22:14:05.0362269Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0362563Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0362833Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:05.0363046Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51871 2023-01-11T22:14:05.0363261Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51872 2023-01-11T22:14:05.0363625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0363794Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0364165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0364355Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0364723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0364876Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0365246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0365431Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0365761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:05.0365999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:05.0366399Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0366795Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0367022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:05.0367245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:05.0367559Z STAGE:2023-01-11 22:12:49 51872:51872 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0367877Z STAGE:2023-01-11 22:12:49 51871:51871 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2023-01-11T22:14:05.0368215Z STAGE:2023-01-11 22:12:49 51871:51871 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0368555Z STAGE:2023-01-11 22:12:49 51871:51871 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0368884Z STAGE:2023-01-11 22:12:49 51872:51872 ActivityProfilerController.cpp:306] Completed Stage: Collection 2023-01-11T22:14:05.0369270Z STAGE:2023-01-11 22:12:49 51872:51872 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2023-01-11T22:14:05.0369378Z ok (4.140s) 2023-01-11T22:14:05.0369398Z 2023-01-11T22:14:05.0369666Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0369762Z Ran 1 test in 4.140s 2023-01-11T22:14:05.0369798Z 2023-01-11T22:14:05.0369873Z OK 2023-01-11T22:14:05.0369892Z 2023-01-11T22:14:05.0370015Z Generating XML reports... 2023-01-11T22:14:05.0370666Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221245.xml 2023-01-11T22:14:05.0371054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0371228Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0371601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0371791Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0371812Z 2023-01-11T22:14:05.0371921Z Running tests... 2023-01-11T22:14:05.0372167Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0372480Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0372739Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:05.0372962Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51984 2023-01-11T22:14:05.0373174Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51985 2023-01-11T22:14:05.0373539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0373712Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0374092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0374278Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0374626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0374801Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0375173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0375447Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0375689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:05.0375932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:05.0376332Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0376725Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0376932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:05.0377158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:05.0377260Z ok (4.250s) 2023-01-11T22:14:05.0377283Z 2023-01-11T22:14:05.0377549Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0377660Z Ran 1 test in 4.251s 2023-01-11T22:14:05.0377680Z 2023-01-11T22:14:05.0377772Z OK 2023-01-11T22:14:05.0377791Z 2023-01-11T22:14:05.0377913Z Generating XML reports... 2023-01-11T22:14:05.0378358Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221252.xml 2023-01-11T22:14:05.0378789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0378954Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0379331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0379518Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0379537Z 2023-01-11T22:14:05.0379648Z Running tests... 2023-01-11T22:14:05.0379910Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0380218Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0380481Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:05.0380698Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52183 2023-01-11T22:14:05.0380894Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52184 2023-01-11T22:14:05.0381264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0381435Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0381807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0381993Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0382358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0382529Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0382903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0383091Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0383317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:05.0383560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:05.0383956Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0384348Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0384683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:05.0384905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:05.0385004Z ok (5.759s) 2023-01-11T22:14:05.0385023Z 2023-01-11T22:14:05.0385288Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0385386Z Ran 1 test in 5.760s 2023-01-11T22:14:05.0385424Z 2023-01-11T22:14:05.0385498Z OK 2023-01-11T22:14:05.0385517Z 2023-01-11T22:14:05.0385634Z Generating XML reports... 2023-01-11T22:14:05.0386078Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221259.xml 2023-01-11T22:14:05.0386445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0386621Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0386996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0387184Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0387203Z 2023-01-11T22:14:05.0387311Z Running tests... 2023-01-11T22:14:05.0387556Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0387914Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0388185Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:05.0388397Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52384 2023-01-11T22:14:05.0388608Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52385 2023-01-11T22:14:05.0388975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0389150Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0389520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0389704Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0390055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0390221Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0390591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0390774Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0391010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:05.0391250Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:05.0391646Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0392036Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0392249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:05.0392467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:05.0392566Z ok (6.218s) 2023-01-11T22:14:05.0392586Z 2023-01-11T22:14:05.0392845Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0392952Z Ran 1 test in 6.218s 2023-01-11T22:14:05.0392971Z 2023-01-11T22:14:05.0393057Z OK 2023-01-11T22:14:05.0393076Z 2023-01-11T22:14:05.0393245Z Generating XML reports... 2023-01-11T22:14:05.0393685Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221307.xml 2023-01-11T22:14:05.0394043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0394198Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0394572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0394757Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0394776Z 2023-01-11T22:14:05.0394881Z Running tests... 2023-01-11T22:14:05.0395139Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0395447Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0395700Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:05.0395918Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52499 2023-01-11T22:14:05.0396112Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52500 2023-01-11T22:14:05.0396478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0396696Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0397078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0397263Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0397620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0397785Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0398156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0398338Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0398561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:05.0398799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:05.0399195Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0399584Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0399807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:05.0400028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:05.0400132Z ok (4.193s) 2023-01-11T22:14:05.0400152Z 2023-01-11T22:14:05.0400418Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0400512Z Ran 1 test in 4.193s 2023-01-11T22:14:05.0400547Z 2023-01-11T22:14:05.0400622Z OK 2023-01-11T22:14:05.0400641Z 2023-01-11T22:14:05.0400763Z Generating XML reports... 2023-01-11T22:14:05.0401211Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221316.xml 2023-01-11T22:14:05.0401573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0401745Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0402118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0402302Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0402375Z 2023-01-11T22:14:05.0402487Z Running tests... 2023-01-11T22:14:05.0402734Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0403042Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0403289Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:05.0403505Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52612 2023-01-11T22:14:05.0403718Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52613 2023-01-11T22:14:05.0404084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0404255Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0404629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0404803Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0405165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0405337Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0405772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0405964Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0406204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:05.0406445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:05.0406840Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0407238Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0407445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:05.0407664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:05.0407766Z ok (5.719s) 2023-01-11T22:14:05.0407785Z 2023-01-11T22:14:05.0408049Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0408161Z Ran 1 test in 5.719s 2023-01-11T22:14:05.0408181Z 2023-01-11T22:14:05.0408272Z OK 2023-01-11T22:14:05.0408291Z 2023-01-11T22:14:05.0408413Z Generating XML reports... 2023-01-11T22:14:05.0408856Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221322.xml 2023-01-11T22:14:05.0409215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0409374Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0409744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0409930Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0409949Z 2023-01-11T22:14:05.0410054Z Running tests... 2023-01-11T22:14:05.0410539Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0410868Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0411155Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:05.0411370Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52723 2023-01-11T22:14:05.0411565Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52724 2023-01-11T22:14:05.0412021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0412191Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0412561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0412752Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0413111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0413282Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0413652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0413835Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0414061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:05.0414303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:05.0414700Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0415150Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0415380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:05.0415600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:05.0416365Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-01-11T22:14:05.0417132Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-01-11T22:14:05.0417233Z ok (6.220s) 2023-01-11T22:14:05.0417254Z 2023-01-11T22:14:05.0417521Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0417635Z Ran 1 test in 6.220s 2023-01-11T22:14:05.0417655Z 2023-01-11T22:14:05.0417746Z OK 2023-01-11T22:14:05.0417765Z 2023-01-11T22:14:05.0417870Z Generating XML reports... 2023-01-11T22:14:05.0418318Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221331.xml 2023-01-11T22:14:05.0418688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0418858Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0419233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0419424Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0419444Z 2023-01-11T22:14:05.0419551Z Running tests... 2023-01-11T22:14:05.0419813Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0420183Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0420445Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:05.0420659Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52838 2023-01-11T22:14:05.0420876Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52839 2023-01-11T22:14:05.0421248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0421418Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0421794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0421982Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0422348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0422502Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0422875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0423062Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0423363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:05.0423611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:05.0424007Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0424400Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0424629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:05.0424853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:05.0425070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T22:14:05.0425304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T22:14:05.0425695Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:05.0426079Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:05.0426316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2023-01-11T22:14:05.0426549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2023-01-11T22:14:05.0426935Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2023-01-11T22:14:05.0427316Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2023-01-11T22:14:05.0427419Z ok (10.831s) 2023-01-11T22:14:05.0427439Z 2023-01-11T22:14:05.0427688Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0427802Z Ran 1 test in 10.831s 2023-01-11T22:14:05.0427822Z 2023-01-11T22:14:05.0427910Z OK 2023-01-11T22:14:05.0427929Z 2023-01-11T22:14:05.0428053Z Generating XML reports... 2023-01-11T22:14:05.0428501Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221339.xml 2023-01-11T22:14:05.0428868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0429093Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0429472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0429659Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0429679Z 2023-01-11T22:14:05.0429770Z Running tests... 2023-01-11T22:14:05.0430033Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0430343Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2023-01-11T22:14:05.0430627Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:14:05.0430842Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52961 2023-01-11T22:14:05.0431051Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52962 2023-01-11T22:14:05.0431422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0431595Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0431952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0432140Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0432559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:14:05.0432736Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:14:05.0433106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:14:05.0433292Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:14:05.0433529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:14:05.0433771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:14:05.0434167Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0434541Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:14:05.0434769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:14:05.0434993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:14:05.0435227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T22:14:05.0435461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T22:14:05.0435854Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:05.0436234Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:14:05.0436470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2023-01-11T22:14:05.0436709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2023-01-11T22:14:05.0437075Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2023-01-11T22:14:05.0437460Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2023-01-11T22:14:05.0437560Z ok (10.882s) 2023-01-11T22:14:05.0437580Z 2023-01-11T22:14:05.0437843Z ---------------------------------------------------------------------- 2023-01-11T22:14:05.0438010Z Ran 1 test in 10.882s 2023-01-11T22:14:05.0438030Z 2023-01-11T22:14:05.0438118Z OK 2023-01-11T22:14:05.0438137Z 2023-01-11T22:14:05.0438258Z Generating XML reports... 2023-01-11T22:14:05.0438705Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221353.xml 2023-01-11T22:14:05.0438724Z 2023-01-11T22:14:05.0439176Z ##[endgroup] 2023-01-11T22:14:05.0439649Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_rj7_8efa) 2023-01-11T22:14:05.0439669Z 2023-01-11T22:14:05.0439794Z Shard 1: ucc should be run in 3 2023-01-11T22:14:05.0440059Z Running distributed/rpc/test_faulty_agent ... [2023-01-11 22:14:04.784367] 2023-01-11T22:14:05.0440558Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/rpc/test_faulty_agent.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:14:04.784598] 2023-01-11T22:14:07.4031311Z 2023-01-11T22:14:07.4032238Z Expand the folded group to see the log file of distributed/rpc/test_faulty_agent 2023-01-11T22:14:07.4033479Z ##[group]PRINTING LOG FILE of distributed/rpc/test_faulty_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-test_faulty_agent_20_d79sp) 2023-01-11T22:14:07.4034101Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph6ipzhh1 2023-01-11T22:14:07.4035095Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph6ipzhh1/_remote_module_non_scriptable.py 2023-01-11T22:14:07.4035438Z 2023-01-11T22:14:07.4035797Z ##[endgroup] 2023-01-11T22:14:07.4036652Z FINISHED PRINTING LOG FILE of distributed/rpc/test_faulty_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-test_faulty_agent_20_d79sp) 2023-01-11T22:14:07.4037016Z 2023-01-11T22:14:07.4037526Z Running distributed/pipeline/sync/test_stream ... [2023-01-11 22:14:07.403181] 2023-01-11T22:14:07.4038148Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_stream.py', '-v'] ... [2023-01-11 22:14:07.403417] 2023-01-11T22:14:12.8381570Z 2023-01-11T22:14:12.8382129Z Expand the folded group to see the log file of distributed/pipeline/sync/test_stream 2023-01-11T22:14:12.8383541Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_stream (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_stream_c_0c1y_8) 2023-01-11T22:14:12.8384097Z ============================= test session starts ============================== 2023-01-11T22:14:12.8384677Z platform linux -- Python 3.10.8, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2023-01-11T22:14:12.8385045Z cachedir: .pytest_cache 2023-01-11T22:14:12.8385576Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2023-01-11T22:14:12.8386044Z torch: 2.0.0a0+git8419ddd 2023-01-11T22:14:12.8386340Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2023-01-11T22:14:12.8386902Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0 2023-01-11T22:14:12.8387309Z collecting ... collected 19 items 2023-01-11T22:14:12.8389786Z Running 19 items in this shard: test/distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cpu, test/distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cuda, test/distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cpu, test/distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cuda, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cpu, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cuda, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cpu, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cuda, test/distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_shifted_view 2023-01-11T22:14:12.8392522Z 2023-01-11T22:14:12.8392758Z distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cpu PASSED [ 5%] 2023-01-11T22:14:12.8393226Z distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cuda PASSED [ 10%] 2023-01-11T22:14:12.8393712Z distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cpu PASSED [ 15%] 2023-01-11T22:14:12.8394221Z distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cuda PASSED [ 21%] 2023-01-11T22:14:12.8394804Z distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cpu PASSED [ 26%] 2023-01-11T22:14:12.8395294Z distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cuda PASSED [ 31%] 2023-01-11T22:14:12.8395779Z distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cpu PASSED [ 36%] 2023-01-11T22:14:12.8396255Z distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cuda PASSED [ 42%] 2023-01-11T22:14:12.8396762Z distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cpu PASSED [ 47%] 2023-01-11T22:14:12.8397184Z distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cuda PASSED [ 52%] 2023-01-11T22:14:12.8397655Z distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cpu PASSED [ 57%] 2023-01-11T22:14:12.8398122Z distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cuda PASSED [ 63%] 2023-01-11T22:14:12.8398583Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cpu PASSED [ 68%] 2023-01-11T22:14:12.8399093Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cuda PASSED [ 73%] 2023-01-11T22:14:12.8399561Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cpu PASSED [ 78%] 2023-01-11T22:14:12.8400055Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cuda PASSED [ 84%] 2023-01-11T22:14:12.8400529Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cpu PASSED [ 89%] 2023-01-11T22:14:12.8401018Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cuda PASSED [ 94%] 2023-01-11T22:14:12.8401525Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_shifted_view PASSED [100%] 2023-01-11T22:14:12.8401806Z 2023-01-11T22:14:12.8401961Z ============================== 19 passed in 2.51s ============================== 2023-01-11T22:14:12.8402139Z 2023-01-11T22:14:12.8402447Z ##[endgroup] 2023-01-11T22:14:12.8403084Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_stream (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_stream_c_0c1y_8) 2023-01-11T22:14:12.8403465Z 2023-01-11T22:14:12.8403737Z Running distributed/pipeline/sync/test_phony ... [2023-01-11 22:14:12.838147] 2023-01-11T22:14:12.8404315Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_phony.py', '-v'] ... [2023-01-11 22:14:12.838429] 2023-01-11T22:14:15.2836169Z 2023-01-11T22:14:15.2836939Z Expand the folded group to see the log file of distributed/pipeline/sync/test_phony 2023-01-11T22:14:15.2838265Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_phony (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_phony_qtkk7fu2) 2023-01-11T22:14:15.2838790Z ============================= test session starts ============================== 2023-01-11T22:14:15.2839396Z platform linux -- Python 3.10.8, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2023-01-11T22:14:15.2839784Z cachedir: .pytest_cache 2023-01-11T22:14:15.2840348Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2023-01-11T22:14:15.2840773Z torch: 2.0.0a0+git8419ddd 2023-01-11T22:14:15.2841095Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2023-01-11T22:14:15.2841662Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0 2023-01-11T22:14:15.2842042Z collecting ... collected 4 items 2023-01-11T22:14:15.2842712Z Running 4 items in this shard: test/distributed/pipeline/sync/test_phony.py::test_phony_size, test/distributed/pipeline/sync/test_phony.py::test_phony_requires_grad, test/distributed/pipeline/sync/test_phony.py::test_cached_phony, test/distributed/pipeline/sync/test_phony.py::test_phony_in_autograd_function 2023-01-11T22:14:15.2843242Z 2023-01-11T22:14:15.2843449Z distributed/pipeline/sync/test_phony.py::test_phony_size PASSED [ 25%] 2023-01-11T22:14:15.2843997Z distributed/pipeline/sync/test_phony.py::test_phony_requires_grad PASSED [ 50%] 2023-01-11T22:14:15.2844475Z distributed/pipeline/sync/test_phony.py::test_cached_phony PASSED [ 75%] 2023-01-11T22:14:15.2844914Z distributed/pipeline/sync/test_phony.py::test_phony_in_autograd_function PASSED [100%] 2023-01-11T22:14:15.2845174Z 2023-01-11T22:14:15.2845328Z ============================== 4 passed in 0.06s =============================== 2023-01-11T22:14:15.2845518Z 2023-01-11T22:14:15.2845809Z ##[endgroup] 2023-01-11T22:14:15.2846434Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_phony (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_phony_qtkk7fu2) 2023-01-11T22:14:15.2846812Z 2023-01-11T22:14:15.2847101Z Running distributed/pipeline/sync/test_dependency ... [2023-01-11 22:14:15.283655] 2023-01-11T22:14:15.2847724Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_dependency.py', '-v'] ... [2023-01-11 22:14:15.283892] 2023-01-11T22:14:19.1193530Z 2023-01-11T22:14:19.1194081Z Expand the folded group to see the log file of distributed/pipeline/sync/test_dependency 2023-01-11T22:14:19.1195211Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_dependency (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_dependency_hvqg4fnz) 2023-01-11T22:14:19.1196140Z ============================= test session starts ============================== 2023-01-11T22:14:19.1196749Z platform linux -- Python 3.10.8, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2023-01-11T22:14:19.1197119Z cachedir: .pytest_cache 2023-01-11T22:14:19.1197669Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2023-01-11T22:14:19.1198333Z torch: 2.0.0a0+git8419ddd 2023-01-11T22:14:19.1198747Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2023-01-11T22:14:19.1199333Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0 2023-01-11T22:14:19.1199717Z collecting ... collected 6 items 2023-01-11T22:14:19.1200623Z Running 6 items in this shard: test/distributed/pipeline/sync/test_dependency.py::test_fork_join, test/distributed/pipeline/sync/test_dependency.py::test_fork_join_enable_grad, test/distributed/pipeline/sync/test_dependency.py::test_fork_join_no_grad, test/distributed/pipeline/sync/test_dependency.py::test_fork_leak, test/distributed/pipeline/sync/test_dependency.py::test_join_when_fork_not_requires_grad, test/distributed/pipeline/sync/test_dependency.py::test_join_when_fork_requires_grad 2023-01-11T22:14:19.1201652Z 2023-01-11T22:14:19.1201860Z distributed/pipeline/sync/test_dependency.py::test_fork_join PASSED [ 16%] 2023-01-11T22:14:19.1202317Z distributed/pipeline/sync/test_dependency.py::test_fork_join_enable_grad PASSED [ 33%] 2023-01-11T22:14:19.1202780Z distributed/pipeline/sync/test_dependency.py::test_fork_join_no_grad PASSED [ 50%] 2023-01-11T22:14:19.1203213Z distributed/pipeline/sync/test_dependency.py::test_fork_leak PASSED [ 66%] 2023-01-11T22:14:19.1203681Z distributed/pipeline/sync/test_dependency.py::test_join_when_fork_not_requires_grad PASSED [ 83%] 2023-01-11T22:14:19.1204168Z distributed/pipeline/sync/test_dependency.py::test_join_when_fork_requires_grad PASSED [100%] 2023-01-11T22:14:19.1204437Z 2023-01-11T22:14:19.1204594Z ============================== 6 passed in 1.40s =============================== 2023-01-11T22:14:19.1204788Z 2023-01-11T22:14:19.1205092Z ##[endgroup] 2023-01-11T22:14:19.1205751Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_dependency (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_dependency_hvqg4fnz) 2023-01-11T22:14:19.1206144Z 2023-01-11T22:14:19.1206431Z Running distributed/pipeline/sync/test_checkpoint ... [2023-01-11 22:14:19.119347] 2023-01-11T22:14:19.1207156Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_checkpoint.py', '-v'] ... [2023-01-11 22:14:19.119587] 2023-01-11T22:14:22.8980333Z 2023-01-11T22:14:22.8981137Z Expand the folded group to see the log file of distributed/pipeline/sync/test_checkpoint 2023-01-11T22:14:22.8982314Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_checkpoint_q9mj2phm) 2023-01-11T22:14:22.8982868Z ============================= test session starts ============================== 2023-01-11T22:14:22.8983484Z platform linux -- Python 3.10.8, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2023-01-11T22:14:22.8983864Z cachedir: .pytest_cache 2023-01-11T22:14:22.8984774Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2023-01-11T22:14:22.8985305Z torch: 2.0.0a0+git8419ddd 2023-01-11T22:14:22.8985630Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2023-01-11T22:14:22.8986198Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0 2023-01-11T22:14:22.8986598Z collecting ... collected 9 items 2023-01-11T22:14:22.8987905Z Running 9 items in this shard: test/distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cpu], test/distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cuda], test/distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad, test/distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad_with_parameter, test/distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cpu], test/distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cuda], test/distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing, test/distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing_without_checkpoint, test/distributed/pipeline/sync/test_checkpoint.py::test_non_grad_output 2023-01-11T22:14:22.8989096Z 2023-01-11T22:14:22.8989337Z distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cpu] PASSED [ 11%] 2023-01-11T22:14:22.8989812Z distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cuda] PASSED [ 22%] 2023-01-11T22:14:22.8990278Z distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad PASSED [ 33%] 2023-01-11T22:14:22.8990755Z distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad_with_parameter PASSED [ 44%] 2023-01-11T22:14:22.8991229Z distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cpu] PASSED [ 55%] 2023-01-11T22:14:22.8991979Z distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cuda] PASSED [ 66%] 2023-01-11T22:14:22.8992469Z distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing PASSED [ 77%] 2023-01-11T22:14:22.8992989Z distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing_without_checkpoint PASSED [ 88%] 2023-01-11T22:14:22.8993495Z distributed/pipeline/sync/test_checkpoint.py::test_non_grad_output PASSED [100%] 2023-01-11T22:14:22.8993750Z 2023-01-11T22:14:22.8993907Z ============================== 9 passed in 1.39s =============================== 2023-01-11T22:14:22.8994098Z 2023-01-11T22:14:22.8994395Z ##[endgroup] 2023-01-11T22:14:22.8995054Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_checkpoint_q9mj2phm) 2023-01-11T22:14:22.8995453Z 2023-01-11T22:14:22.8995771Z Running distributed/pipeline/sync/skip/test_verify_skippables ... [2023-01-11 22:14:22.898043] 2023-01-11T22:14:22.8996440Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_verify_skippables.py', '-v'] ... [2023-01-11 22:14:22.898306] 2023-01-11T22:14:25.2498951Z 2023-01-11T22:14:25.2499511Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_verify_skippables 2023-01-11T22:14:25.2500802Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_verify_skippables (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_verify_skippables_qn0u3vzh) 2023-01-11T22:14:25.2501610Z ============================= test session starts ============================== 2023-01-11T22:14:25.2502225Z platform linux -- Python 3.10.8, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2023-01-11T22:14:25.2502578Z cachedir: .pytest_cache 2023-01-11T22:14:25.2503119Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2023-01-11T22:14:25.2503567Z torch: 2.0.0a0+git8419ddd 2023-01-11T22:14:25.2503884Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2023-01-11T22:14:25.2504449Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0 2023-01-11T22:14:25.2504842Z collecting ... collected 9 items 2023-01-11T22:14:25.2506146Z Running 9 items in this shard: test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_matching, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_not_pop, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_unknown, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_again, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_again, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_different_names, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_same_name, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop_but_isolated 2023-01-11T22:14:25.2507307Z 2023-01-11T22:14:25.2507533Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_matching PASSED [ 11%] 2023-01-11T22:14:25.2508008Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_not_pop PASSED [ 22%] 2023-01-11T22:14:25.2508480Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_unknown PASSED [ 33%] 2023-01-11T22:14:25.2508929Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_again PASSED [ 44%] 2023-01-11T22:14:25.2509409Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_again PASSED [ 55%] 2023-01-11T22:14:25.2509913Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_different_names PASSED [ 66%] 2023-01-11T22:14:25.2510560Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_same_name PASSED [ 77%] 2023-01-11T22:14:25.2511043Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop PASSED [ 88%] 2023-01-11T22:14:25.2511542Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop_but_isolated PASSED [100%] 2023-01-11T22:14:25.2511829Z 2023-01-11T22:14:25.2511984Z ============================== 9 passed in 0.05s =============================== 2023-01-11T22:14:25.2512178Z 2023-01-11T22:14:25.2512475Z ##[endgroup] 2023-01-11T22:14:25.2513188Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_verify_skippables (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_verify_skippables_qn0u3vzh) 2023-01-11T22:14:25.2513626Z 2023-01-11T22:14:25.2513912Z Running distributed/pipeline/sync/skip/test_portal ... [2023-01-11 22:14:25.250005] 2023-01-11T22:14:25.2514526Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_portal.py', '-v'] ... [2023-01-11 22:14:25.250324] 2023-01-11T22:14:29.1073126Z 2023-01-11T22:14:29.1073834Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_portal 2023-01-11T22:14:29.1074800Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_portal (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_portal_xpnp470x) 2023-01-11T22:14:29.1075837Z ============================= test session starts ============================== 2023-01-11T22:14:29.1076494Z platform linux -- Python 3.10.8, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2023-01-11T22:14:29.1076847Z cachedir: .pytest_cache 2023-01-11T22:14:29.1077396Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2023-01-11T22:14:29.1077841Z torch: 2.0.0a0+git8419ddd 2023-01-11T22:14:29.1078331Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2023-01-11T22:14:29.1079025Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0 2023-01-11T22:14:29.1079422Z collecting ... collected 10 items 2023-01-11T22:14:29.1080783Z Running 10 items in this shard: test/distributed/pipeline/sync/skip/test_portal.py::test_copy_returns_on_next_device, test/distributed/pipeline/sync/skip/test_portal.py::test_blue_orange, test/distributed/pipeline/sync/skip/test_portal.py::test_blue_orange_not_requires_grad, test/distributed/pipeline/sync/skip/test_portal.py::test_use_grad, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_0, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_1, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_2, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_4, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3_plus_1 2023-01-11T22:14:29.1081991Z 2023-01-11T22:14:29.1082222Z distributed/pipeline/sync/skip/test_portal.py::test_copy_returns_on_next_device PASSED [ 10%] 2023-01-11T22:14:29.1082683Z distributed/pipeline/sync/skip/test_portal.py::test_blue_orange PASSED [ 20%] 2023-01-11T22:14:29.1083142Z distributed/pipeline/sync/skip/test_portal.py::test_blue_orange_not_requires_grad PASSED [ 30%] 2023-01-11T22:14:29.1083579Z distributed/pipeline/sync/skip/test_portal.py::test_use_grad PASSED [ 40%] 2023-01-11T22:14:29.1084031Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_0 PASSED [ 50%] 2023-01-11T22:14:29.1084511Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_1 PASSED [ 60%] 2023-01-11T22:14:29.1084968Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_2 PASSED [ 70%] 2023-01-11T22:14:29.1085441Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3 PASSED [ 80%] 2023-01-11T22:14:29.1086025Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_4 PASSED [ 90%] 2023-01-11T22:14:29.1086512Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3_plus_1 PASSED [100%] 2023-01-11T22:14:29.1086793Z 2023-01-11T22:14:29.1086932Z ============================== 10 passed in 1.41s ============================== 2023-01-11T22:14:29.1087124Z 2023-01-11T22:14:29.1087435Z ##[endgroup] 2023-01-11T22:14:29.1088097Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_portal (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_portal_xpnp470x) 2023-01-11T22:14:29.1088496Z 2023-01-11T22:14:29.1088764Z Running distributed/pipeline/sync/skip/test_gpipe ... [2023-01-11 22:14:29.107326] 2023-01-11T22:14:29.1089393Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_gpipe.py', '-v'] ... [2023-01-11 22:14:29.107652] 2023-01-11T22:14:36.9688414Z 2023-01-11T22:14:36.9689188Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_gpipe 2023-01-11T22:14:36.9690186Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_gpipe (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_gpipe_jyz6htp1) 2023-01-11T22:14:36.9691286Z ============================= test session starts ============================== 2023-01-11T22:14:36.9692139Z platform linux -- Python 3.10.8, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2023-01-11T22:14:36.9692547Z cachedir: .pytest_cache 2023-01-11T22:14:36.9693121Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2023-01-11T22:14:36.9693563Z torch: 2.0.0a0+git8419ddd 2023-01-11T22:14:36.9693915Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2023-01-11T22:14:36.9694748Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0 2023-01-11T22:14:36.9695174Z collecting ... collected 13 items 2023-01-11T22:14:36.9697371Z Running 13 items in this shard: test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-3], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:2], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-2:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:1:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-3], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:2], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-2:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:1:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-3], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:2], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-2:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:1:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_none_skip 2023-01-11T22:14:36.9698717Z 2023-01-11T22:14:36.9699000Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-3] PASSED [ 7%] 2023-01-11T22:14:36.9699527Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:2] PASSED [ 15%] 2023-01-11T22:14:36.9700042Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-2:1] PASSED [ 23%] 2023-01-11T22:14:36.9700580Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:1:1] SKIPPED [ 30%] 2023-01-11T22:14:36.9701094Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-3] PASSED [ 38%] 2023-01-11T22:14:36.9701621Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:2] PASSED [ 46%] 2023-01-11T22:14:36.9702142Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-2:1] PASSED [ 53%] 2023-01-11T22:14:36.9702664Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:1:1] SKIPPED [ 61%] 2023-01-11T22:14:36.9703330Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-3] PASSED [ 69%] 2023-01-11T22:14:36.9703881Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:2] PASSED [ 76%] 2023-01-11T22:14:36.9704409Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-2:1] PASSED [ 84%] 2023-01-11T22:14:36.9704941Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:1:1] SKIPPED [ 92%] 2023-01-11T22:14:36.9705389Z distributed/pipeline/sync/skip/test_gpipe.py::test_none_skip PASSED [100%] 2023-01-11T22:14:36.9705638Z 2023-01-11T22:14:36.9705805Z ======================== 10 passed, 3 skipped in 5.03s ========================= 2023-01-11T22:14:36.9706011Z 2023-01-11T22:14:36.9706309Z ##[endgroup] 2023-01-11T22:14:36.9706959Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_gpipe (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_gpipe_jyz6htp1) 2023-01-11T22:14:36.9707374Z 2023-01-11T22:14:36.9707673Z Running distributed/optim/test_apply_optimizer_in_backward ... [2023-01-11 22:14:36.968854] 2023-01-11T22:14:36.9708414Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/optim/test_apply_optimizer_in_backward.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:14:36.969227] 2023-01-11T22:14:39.1180498Z 2023-01-11T22:14:39.1181333Z Expand the folded group to see the log file of distributed/optim/test_apply_optimizer_in_backward 2023-01-11T22:14:39.1182880Z ##[group]PRINTING LOG FILE of distributed/optim/test_apply_optimizer_in_backward (/var/lib/jenkins/workspace/test/test-reports/distributed-optim-test_apply_optimizer_in_backward_dnineaes) 2023-01-11T22:14:39.1183342Z 2023-01-11T22:14:39.1183639Z ##[endgroup] 2023-01-11T22:14:39.1184636Z FINISHED PRINTING LOG FILE of distributed/optim/test_apply_optimizer_in_backward (/var/lib/jenkins/workspace/test/test-reports/distributed-optim-test_apply_optimizer_in_backward_dnineaes) 2023-01-11T22:14:39.1185067Z 2023-01-11T22:14:39.1185566Z Running distributed/elastic/events/lib_test ... [2023-01-11 22:14:39.118108] 2023-01-11T22:14:39.1186693Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/elastic/events/lib_test.py', '-v'] ... [2023-01-11 22:14:39.118391] 2023-01-11T22:14:41.6383982Z 2023-01-11T22:14:41.6385075Z Expand the folded group to see the log file of distributed/elastic/events/lib_test 2023-01-11T22:14:41.6386845Z ##[group]PRINTING LOG FILE of distributed/elastic/events/lib_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-events-lib_test_5oioz6vb) 2023-01-11T22:14:41.6387382Z ============================= test session starts ============================== 2023-01-11T22:14:41.6387993Z platform linux -- Python 3.10.8, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2023-01-11T22:14:41.6388372Z cachedir: .pytest_cache 2023-01-11T22:14:41.6388906Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2023-01-11T22:14:41.6389441Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2023-01-11T22:14:41.6389995Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0 2023-01-11T22:14:41.6390401Z collecting ... collected 8 items 2023-01-11T22:14:41.6391619Z Running 8 items in this shard: test/distributed/elastic/events/lib_test.py::EventLibTest::test_event_created, test/distributed/elastic/events/lib_test.py::EventLibTest::test_event_deser, test/distributed/elastic/events/lib_test.py::EventLibTest::test_get_or_create_logger, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event_does_not_run_if_invalid_dest, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_created, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_deserialize, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_str 2023-01-11T22:14:41.6392990Z 2023-01-11T22:14:41.6393196Z distributed/elastic/events/lib_test.py::EventLibTest::test_event_created PASSED [ 12%] 2023-01-11T22:14:41.6393655Z distributed/elastic/events/lib_test.py::EventLibTest::test_event_deser PASSED [ 25%] 2023-01-11T22:14:41.6394113Z distributed/elastic/events/lib_test.py::EventLibTest::test_get_or_create_logger PASSED [ 37%] 2023-01-11T22:14:41.6394611Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event PASSED [ 50%] 2023-01-11T22:14:41.6395150Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event_does_not_run_if_invalid_dest PASSED [ 62%] 2023-01-11T22:14:41.6395688Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_created PASSED [ 75%] 2023-01-11T22:14:41.6396177Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_deserialize PASSED [ 87%] 2023-01-11T22:14:41.6396688Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_str PASSED [100%] 2023-01-11T22:14:41.6396915Z 2023-01-11T22:14:41.6397069Z ============================== 8 passed in 1.77s =============================== 2023-01-11T22:14:41.6397261Z 2023-01-11T22:14:41.6397568Z ##[endgroup] 2023-01-11T22:14:41.6398286Z FINISHED PRINTING LOG FILE of distributed/elastic/events/lib_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-events-lib_test_5oioz6vb) 2023-01-11T22:14:41.6398644Z 2023-01-11T22:14:41.6398932Z Running distributed/_shard/test_replicated_tensor ... [2023-01-11 22:14:41.638500] 2023-01-11T22:14:41.6399640Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/test_replicated_tensor.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:14:41.638808] 2023-01-11T22:14:43.7699599Z 2023-01-11T22:14:43.7700384Z Expand the folded group to see the log file of distributed/_shard/test_replicated_tensor 2023-01-11T22:14:43.7701357Z ##[group]PRINTING LOG FILE of distributed/_shard/test_replicated_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_replicated_tensor_y75weal4) 2023-01-11T22:14:43.7701730Z 2023-01-11T22:14:43.7702135Z ##[endgroup] 2023-01-11T22:14:43.7702955Z FINISHED PRINTING LOG FILE of distributed/_shard/test_replicated_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_replicated_tensor_y75weal4) 2023-01-11T22:14:43.7703525Z 2023-01-11T22:14:43.7703862Z Running distributed/_composable/test_checkpoint ... [2023-01-11 22:14:43.770022] 2023-01-11T22:14:43.7706506Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_composable/test_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:14:43.770332] 2023-01-11T22:14:48.1782701Z 2023-01-11T22:14:48.1783881Z Expand the folded group to see the log file of distributed/_composable/test_checkpoint 2023-01-11T22:14:48.1785682Z ##[group]PRINTING LOG FILE of distributed/_composable/test_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_checkpoint_zpz4hg4g) 2023-01-11T22:14:48.1786399Z 2023-01-11T22:14:48.1786583Z Running tests... 2023-01-11T22:14:48.1787475Z ---------------------------------------------------------------------- 2023-01-11T22:14:48.1788508Z Test results will be stored in test-reports/python-unittest/distributed._composable.test_checkpoint 2023-01-11T22:14:48.1789185Z test_random_cpu (__main__.TestCheckpoint) ... ok (0.019s) 2023-01-11T22:14:48.1789570Z test_tensor_only_cpu_use_reentrant_False (__main__.TestCheckpoint) ... ok (0.005s) 2023-01-11T22:14:48.1789999Z test_tensor_only_cpu_use_reentrant_True (__main__.TestCheckpoint) ... ok (0.005s) 2023-01-11T22:14:48.1790422Z test_tensor_only_gpu_use_reentrant_False (__main__.TestCheckpoint) ... ok (0.420s) 2023-01-11T22:14:48.1790823Z test_tensor_only_gpu_use_reentrant_True (__main__.TestCheckpoint) ... ok (0.006s) 2023-01-11T22:14:48.1791065Z 2023-01-11T22:14:48.1791809Z ---------------------------------------------------------------------- 2023-01-11T22:14:48.1792350Z Ran 5 tests in 0.456s 2023-01-11T22:14:48.1792517Z 2023-01-11T22:14:48.1792611Z OK 2023-01-11T22:14:48.1792726Z 2023-01-11T22:14:48.1792849Z Generating XML reports... 2023-01-11T22:14:48.1793473Z Generated XML report: test-reports/python-unittest/distributed._composable.test_checkpoint/TEST-TestCheckpoint-20230111221447.xml 2023-01-11T22:14:48.1793826Z 2023-01-11T22:14:48.1794159Z ##[endgroup] 2023-01-11T22:14:48.1794783Z FINISHED PRINTING LOG FILE of distributed/_composable/test_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_checkpoint_zpz4hg4g) 2023-01-11T22:14:48.1795155Z 2023-01-11T22:14:48.1795430Z Running distributed/_tools/test_memory_tracker ... [2023-01-11 22:14:48.178304] 2023-01-11T22:14:48.1796113Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tools/test_memory_tracker.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:14:48.178641] 2023-01-11T22:14:53.5731891Z 2023-01-11T22:14:53.5733149Z Expand the folded group to see the log file of distributed/_tools/test_memory_tracker 2023-01-11T22:14:53.5734708Z ##[group]PRINTING LOG FILE of distributed/_tools/test_memory_tracker (/var/lib/jenkins/workspace/test/test-reports/distributed-_tools-test_memory_tracker_dhdhttt9) 2023-01-11T22:14:53.5735380Z 2023-01-11T22:14:53.5735554Z Running tests... 2023-01-11T22:14:53.5736704Z ---------------------------------------------------------------------- 2023-01-11T22:14:53.5737821Z Test results will be stored in test-reports/python-unittest/distributed._tools.test_memory_tracker 2023-01-11T22:14:53.5738599Z test_local_model (__main__.TestMemoryTracker) 2023-01-11T22:14:53.5739278Z Minimal test case to check the memory tracker can collect the expected ... ok (1.356s) 2023-01-11T22:14:53.5739757Z ------------------------------------------------ 2023-01-11T22:14:53.5740081Z Top 20 ops that generates memory are: 2023-01-11T22:14:53.5740398Z 0.1.forward.cudnn_batch_norm.default_0: 24.5009765625MB 2023-01-11T22:14:53.5740733Z 0.0.forward.convolution.default_0: 24.5MB 2023-01-11T22:14:53.5741046Z 0.2.forward.relu.default_0: 24.5MB 2023-01-11T22:14:53.5741323Z 2.1.forward.div.Scalar_0: 24.5MB 2023-01-11T22:14:53.5741722Z 2.1.forward.threshold_backward.default_1: 24.49951171875MB 2023-01-11T22:14:53.5742326Z 2.0.forward.addmm.default_0: 4.00048828125MB 2023-01-11T22:14:53.5742695Z 2.1.forward.mm.default_0: 4.00048828125MB 2023-01-11T22:14:53.5743023Z 2.1.forward.nll_loss_forward.default_0: 0.0009765625MB 2023-01-11T22:14:53.5743334Z ._to_copy.default_0: 0.00048828125MB 2023-01-11T22:14:53.5743625Z 2.1.forward._log_softmax.default_0: 0.00048828125MB 2023-01-11T22:14:53.5743944Z 2.1.forward.ones_like.default_0: 0.00048828125MB 2023-01-11T22:14:53.5744273Z 2.1.forward.nll_loss_backward.default_0: 0.00048828125MB 2023-01-11T22:14:53.5744589Z 2.1.forward.mm.default_1: 0.00048828125MB 2023-01-11T22:14:53.5744877Z 2.1.forward.sum.dim_IntList_0: 0.00048828125MB 2023-01-11T22:14:53.5745171Z .lift_fresh.default_0: 0.0MB 2023-01-11T22:14:53.5745481Z 0.1.forward.add_.Tensor_0: 0.0MB 2023-01-11T22:14:53.5745747Z 1.forward.view.default_0: 0.0MB 2023-01-11T22:14:53.5746086Z 2.0.forward.t.default_0: 0.0MB 2023-01-11T22:14:53.5746367Z 2.1.forward.relu_.default_0: 0.0MB 2023-01-11T22:14:53.5746670Z 2.1.forward._log_softmax_backward_data.default_0: 0.0MB 2023-01-11T22:14:53.5747088Z ------------------------------------------------ 2023-01-11T22:14:53.5747291Z 2023-01-11T22:14:53.5747550Z ---------------------------------------------------------------------- 2023-01-11T22:14:53.5747876Z Ran 1 test in 1.356s 2023-01-11T22:14:53.5748018Z 2023-01-11T22:14:53.5748110Z OK 2023-01-11T22:14:53.5748243Z 2023-01-11T22:14:53.5748365Z Generating XML reports... 2023-01-11T22:14:53.5748973Z Generated XML report: test-reports/python-unittest/distributed._tools.test_memory_tracker/TEST-TestMemoryTracker-20230111221451.xml 2023-01-11T22:14:53.5749327Z 2023-01-11T22:14:53.5749816Z ##[endgroup] 2023-01-11T22:14:53.5750422Z FINISHED PRINTING LOG FILE of distributed/_tools/test_memory_tracker (/var/lib/jenkins/workspace/test/test-reports/distributed-_tools-test_memory_tracker_dhdhttt9) 2023-01-11T22:14:53.5750788Z 2023-01-11T22:14:53.5751062Z Running distributed/nn/jit/test_instantiator ... [2023-01-11 22:14:53.573324] 2023-01-11T22:14:53.5751728Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/nn/jit/test_instantiator.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:14:53.573670] 2023-01-11T22:14:57.4728845Z 2023-01-11T22:14:57.4729652Z Expand the folded group to see the log file of distributed/nn/jit/test_instantiator 2023-01-11T22:14:57.4731804Z ##[group]PRINTING LOG FILE of distributed/nn/jit/test_instantiator (/var/lib/jenkins/workspace/test/test-reports/distributed-nn-jit-test_instantiator_i4zg0djq) 2023-01-11T22:14:57.4732181Z 2023-01-11T22:14:57.4732298Z Running tests... 2023-01-11T22:14:57.4732778Z ---------------------------------------------------------------------- 2023-01-11T22:14:57.4733368Z Test results will be stored in test-reports/python-unittest/distributed.nn.jit.test_instantiator 2023-01-11T22:14:57.4733839Z test_get_arg_return_types_from_interface (__main__.TestInstantiator) ... ok (1.631s) 2023-01-11T22:14:57.4734301Z test_instantiate_non_scripted_remote_module_template (__main__.TestInstantiator) ... ok (0.002s) 2023-01-11T22:14:57.4734998Z test_instantiate_scripted_remote_module_template (__main__.TestInstantiator) ... ok (0.014s) 2023-01-11T22:14:57.4735277Z 2023-01-11T22:14:57.4735548Z ---------------------------------------------------------------------- 2023-01-11T22:14:57.4735877Z Ran 3 tests in 1.647s 2023-01-11T22:14:57.4736041Z 2023-01-11T22:14:57.4736117Z OK 2023-01-11T22:14:57.4736251Z 2023-01-11T22:14:57.4736374Z Generating XML reports... 2023-01-11T22:14:57.4736986Z Generated XML report: test-reports/python-unittest/distributed.nn.jit.test_instantiator/TEST-TestInstantiator-20230111221455.xml 2023-01-11T22:14:57.4737346Z 2023-01-11T22:14:57.4737645Z ##[endgroup] 2023-01-11T22:14:57.4738249Z FINISHED PRINTING LOG FILE of distributed/nn/jit/test_instantiator (/var/lib/jenkins/workspace/test/test-reports/distributed-nn-jit-test_instantiator_i4zg0djq) 2023-01-11T22:14:57.4738626Z 2023-01-11T22:14:57.4738874Z Running distributed/test_launcher ... [2023-01-11 22:14:57.472921] 2023-01-11T22:14:57.4739522Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_launcher.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:14:57.473280] 2023-01-11T22:15:01.8306813Z 2023-01-11T22:15:01.8307594Z Expand the folded group to see the log file of distributed/test_launcher 2023-01-11T22:15:01.8308969Z ##[group]PRINTING LOG FILE of distributed/test_launcher (/var/lib/jenkins/workspace/test/test-reports/distributed-test_launcher_skj6x_rf) 2023-01-11T22:15:01.8309420Z 2023-01-11T22:15:01.8309626Z Running tests... 2023-01-11T22:15:01.8310699Z ---------------------------------------------------------------------- 2023-01-11T22:15:01.8311638Z Test results will be stored in test-reports/python-unittest/distributed.test_launcher 2023-01-11T22:15:01.8313053Z test_launch_user_script (__main__.TestDistributedLaunch) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/79488 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.658s) 2023-01-11T22:15:01.8313699Z 2023-01-11T22:15:01.8313955Z ---------------------------------------------------------------------- 2023-01-11T22:15:01.8314287Z Ran 1 test in 1.658s 2023-01-11T22:15:01.8314457Z 2023-01-11T22:15:01.8314569Z OK (skipped=1) 2023-01-11T22:15:01.8314726Z 2023-01-11T22:15:01.8314870Z Generating XML reports... 2023-01-11T22:15:01.8315448Z Generated XML report: test-reports/python-unittest/distributed.test_launcher/TEST-TestDistributedLaunch-20230111221459.xml 2023-01-11T22:15:01.8316086Z 2023-01-11T22:15:01.8316413Z ##[endgroup] 2023-01-11T22:15:01.8316973Z FINISHED PRINTING LOG FILE of distributed/test_launcher (/var/lib/jenkins/workspace/test/test-reports/distributed-test_launcher_skj6x_rf) 2023-01-11T22:15:01.8317289Z 2023-01-11T22:15:01.8317548Z Running distributed/fsdp/test_utils ... [2023-01-11 22:15:01.830724] 2023-01-11T22:15:01.8318207Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_utils.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:15:01.831056] 2023-01-11T22:15:05.8495383Z 2023-01-11T22:15:05.8495916Z Expand the folded group to see the log file of distributed/fsdp/test_utils 2023-01-11T22:15:05.8496945Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_utils (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_utils_l55ovham) 2023-01-11T22:15:05.8497574Z 2023-01-11T22:15:05.8497699Z Running tests... 2023-01-11T22:15:05.8498188Z ---------------------------------------------------------------------- 2023-01-11T22:15:05.8498752Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_utils 2023-01-11T22:15:05.8499302Z test_get_fully_sharded_module_to_states (__main__.TestGetSubmoduleToStates) 2023-01-11T22:15:05.8500037Z Tests the helper function ``_get_fully_sharded_module_states()`` that ... ok (1.670s) 2023-01-11T22:15:05.8500448Z test_apply_to_tensors_cpu_cuda (__main__.TestUtils) ... ok (0.003s) 2023-01-11T22:15:05.8501199Z test_apply_to_tensors_devices_['cpu'] (__main__.TestUtils) ... ok (0.003s) 2023-01-11T22:15:05.8501700Z test_apply_to_tensors_devices_['cuda'] (__main__.TestUtils) ... ok (0.003s) 2023-01-11T22:15:05.8502039Z test_packed_sequence (__main__.TestUtils) 2023-01-11T22:15:05.8502408Z Test to ensure RNN packed sequences are modified correctly. ... ok (0.002s) 2023-01-11T22:15:05.8502814Z test_replace_by_prefix (__main__.TestUtils) ... ok (0.001s) 2023-01-11T22:15:05.8503025Z 2023-01-11T22:15:05.8503275Z ---------------------------------------------------------------------- 2023-01-11T22:15:05.8503610Z Ran 6 tests in 1.684s 2023-01-11T22:15:05.8503773Z 2023-01-11T22:15:05.8503868Z OK 2023-01-11T22:15:05.8504004Z 2023-01-11T22:15:05.8504127Z Generating XML reports... 2023-01-11T22:15:05.8504720Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestGetSubmoduleToStates-20230111221503.xml 2023-01-11T22:15:05.8505461Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestUtils-20230111221503.xml 2023-01-11T22:15:05.8505781Z 2023-01-11T22:15:05.8506098Z ##[endgroup] 2023-01-11T22:15:05.8506643Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_utils (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_utils_l55ovham) 2023-01-11T22:15:05.8506983Z 2023-01-11T22:15:05.8507244Z Running distributed/fsdp/test_fsdp_fx ... [2023-01-11 22:15:05.849564] 2023-01-11T22:15:05.8507900Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_fx.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:15:05.849813] 2023-01-11T22:15:09.8684030Z 2023-01-11T22:15:09.8684548Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_fx 2023-01-11T22:15:09.8685483Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_fx (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_fx_4du6ic1v) 2023-01-11T22:15:09.8685827Z 2023-01-11T22:15:09.8685941Z Running tests... 2023-01-11T22:15:09.8686440Z ---------------------------------------------------------------------- 2023-01-11T22:15:09.8686984Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_fx 2023-01-11T22:15:09.8687429Z test_symbolic_tracing_outputs (__main__.TestSymbolicTracing) 2023-01-11T22:15:09.8687872Z Tests running ``tracer.trace()`` inside ``patch_tracer()`` by checking ... ok (1.671s) 2023-01-11T22:15:09.8688091Z 2023-01-11T22:15:09.8688359Z ---------------------------------------------------------------------- 2023-01-11T22:15:09.8688970Z Ran 1 test in 1.671s 2023-01-11T22:15:09.8689134Z 2023-01-11T22:15:09.8689229Z OK 2023-01-11T22:15:09.8689363Z 2023-01-11T22:15:09.8689470Z Generating XML reports... 2023-01-11T22:15:09.8690090Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_fx/TEST-TestSymbolicTracing-20230111221507.xml 2023-01-11T22:15:09.8690786Z 2023-01-11T22:15:09.8691110Z ##[endgroup] 2023-01-11T22:15:09.8691682Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_fx (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_fx_4du6ic1v) 2023-01-11T22:15:09.8692024Z 2023-01-11T22:15:09.8692306Z Running distributed/_tensor/test_pointwise_ops ... [2023-01-11 22:15:09.868398] 2023-01-11T22:15:09.8693004Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_pointwise_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:15:09.868668] 2023-01-11T22:15:13.8581824Z 2023-01-11T22:15:13.8582323Z Expand the folded group to see the log file of distributed/_tensor/test_pointwise_ops 2023-01-11T22:15:13.8583241Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_pointwise_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_pointwise_ops_zaa5kar1) 2023-01-11T22:15:13.8583603Z 2023-01-11T22:15:13.8583717Z Running tests... 2023-01-11T22:15:13.8584220Z ---------------------------------------------------------------------- 2023-01-11T22:15:13.8585025Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_pointwise_ops 2023-01-11T22:15:13.8585551Z test_activations (__main__.DistElementwiseOpsTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:15:13.8586038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:15:13.8586541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:15:13.8586995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2023-01-11T22:15:13.8587486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2023-01-11T22:15:13.8588165Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2023-01-11T22:15:13.8588833Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2023-01-11T22:15:13.8589510Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2023-01-11T22:15:13.8590176Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2023-01-11T22:15:13.8590571Z ok (1.642s) 2023-01-11T22:15:13.8591010Z test_dropout (__main__.DistElementwiseOpsTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:15:13.8591591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:15:13.8592074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2023-01-11T22:15:13.8592559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2023-01-11T22:15:13.8593213Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2023-01-11T22:15:13.8593874Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2023-01-11T22:15:13.8594549Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2023-01-11T22:15:13.8595221Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2023-01-11T22:15:13.8595991Z INFO:torch.testing._internal.common_distributed:Thread 2 skipping test test_dropout for following reason: testing RNG based ops is broken: https://github.com/pytorch/tau/issues/494 2023-01-11T22:15:13.8596682Z INFO:torch.testing._internal.common_distributed:Thread 0 skipping test test_dropout for following reason: testing RNG based ops is broken: https://github.com/pytorch/tau/issues/494 2023-01-11T22:15:13.8597391Z INFO:torch.testing._internal.common_distributed:Thread 1 skipping test test_dropout for following reason: testing RNG based ops is broken: https://github.com/pytorch/tau/issues/494 2023-01-11T22:15:13.8598092Z INFO:torch.testing._internal.common_distributed:Thread 3 skipping test test_dropout for following reason: testing RNG based ops is broken: https://github.com/pytorch/tau/issues/494 2023-01-11T22:15:13.8598664Z skip: Test skipped at subprocess level, look at subprocess log for skip reason (0.013s) 2023-01-11T22:15:13.8599227Z test_dropout_backward (__main__.DistElementwiseOpsTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:15:13.8599790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:15:13.8600274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2023-01-11T22:15:13.8600755Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2023-01-11T22:15:13.8601452Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2023-01-11T22:15:13.8602144Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2023-01-11T22:15:13.8602822Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2023-01-11T22:15:13.8603493Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2023-01-11T22:15:13.8603867Z ok (0.022s) 2023-01-11T22:15:13.8604311Z test_dropout_errors (__main__.DistElementwiseOpsTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:15:13.8604874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:15:13.8605360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2023-01-11T22:15:13.8605824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2023-01-11T22:15:13.8606463Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2023-01-11T22:15:13.8607136Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2023-01-11T22:15:13.8607816Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2023-01-11T22:15:13.8608469Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2023-01-11T22:15:13.8608860Z ok (0.013s) 2023-01-11T22:15:13.8609295Z test_mul_out (__main__.DistElementwiseOpsTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:15:13.8609838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:15:13.8610582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2023-01-11T22:15:13.8611073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2023-01-11T22:15:13.8611727Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2023-01-11T22:15:13.8612488Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2023-01-11T22:15:13.8613162Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2023-01-11T22:15:13.8613835Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2023-01-11T22:15:13.8614228Z ok (0.017s) 2023-01-11T22:15:13.8614375Z 2023-01-11T22:15:13.8614626Z ---------------------------------------------------------------------- 2023-01-11T22:15:13.8614954Z Ran 5 tests in 1.707s 2023-01-11T22:15:13.8615117Z 2023-01-11T22:15:13.8615222Z OK (skipped=1) 2023-01-11T22:15:13.8615374Z 2023-01-11T22:15:13.8615480Z Generating XML reports... 2023-01-11T22:15:13.8616097Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_pointwise_ops/TEST-DistElementwiseOpsTest-20230111221511.xml 2023-01-11T22:15:13.8616475Z 2023-01-11T22:15:13.8616791Z ##[endgroup] 2023-01-11T22:15:13.8617374Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_pointwise_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_pointwise_ops_zaa5kar1) 2023-01-11T22:15:13.8617739Z 2023-01-11T22:15:13.8618030Z Running distributed/checkpoint/test_fsdp_optim_state ... [2023-01-11 22:15:13.858302] 2023-01-11T22:15:13.8618814Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_fsdp_optim_state.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:15:13.858563] 2023-01-11T22:15:20.0657997Z 2023-01-11T22:15:20.0658577Z Expand the folded group to see the log file of distributed/checkpoint/test_fsdp_optim_state 2023-01-11T22:15:20.0660055Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_fsdp_optim_state (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_fsdp_optim_state_72wozmva) 2023-01-11T22:15:20.0660844Z 2023-01-11T22:15:20.0661082Z Running tests... 2023-01-11T22:15:20.0662093Z ---------------------------------------------------------------------- 2023-01-11T22:15:20.0662787Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_fsdp_optim_state 2023-01-11T22:15:20.0663303Z test_distributed_tensor_planner (__main__.FsdpOptimStateCheckpoint) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:15:20.0663838Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54305 2023-01-11T22:15:20.0664257Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54306 2023-01-11T22:15:20.0664887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:15:20.0665337Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:15:20.0665877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:15:20.0666351Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:15:20.0666910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:15:20.0667356Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:15:20.0667926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:15:20.0668387Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:15:20.0668802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:15:20.0669285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:15:20.0669760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:15:20.0670213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:15:20.0671144Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:15:20.0671813Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:15:20.0672250Z skip: Need at least 4 CUDA devices (3.900s) 2023-01-11T22:15:20.0672425Z 2023-01-11T22:15:20.0672694Z ---------------------------------------------------------------------- 2023-01-11T22:15:20.0673015Z Ran 1 test in 3.900s 2023-01-11T22:15:20.0673176Z 2023-01-11T22:15:20.0673282Z OK (skipped=1) 2023-01-11T22:15:20.0673435Z 2023-01-11T22:15:20.0673542Z Generating XML reports... 2023-01-11T22:15:20.0674187Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_fsdp_optim_state/TEST-FsdpOptimStateCheckpoint-20230111221515.xml 2023-01-11T22:15:20.0674582Z 2023-01-11T22:15:20.0674907Z ##[endgroup] 2023-01-11T22:15:20.0675534Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_fsdp_optim_state (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_fsdp_optim_state_72wozmva) 2023-01-11T22:15:20.0675933Z 2023-01-11T22:15:20.0676181Z Running distributed/test_data_parallel ... [2023-01-11 22:15:20.065844] 2023-01-11T22:15:20.0676949Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_data_parallel.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:15:20.066213] 2023-01-11T22:15:29.1331603Z 2023-01-11T22:15:29.1332290Z Expand the folded group to see the log file of distributed/test_data_parallel 2023-01-11T22:15:29.1333178Z ##[group]PRINTING LOG FILE of distributed/test_data_parallel (/var/lib/jenkins/workspace/test/test-reports/distributed-test_data_parallel_58c257bf) 2023-01-11T22:15:29.1333557Z 2023-01-11T22:15:29.1333673Z Running tests... 2023-01-11T22:15:29.1334438Z ---------------------------------------------------------------------- 2023-01-11T22:15:29.1334986Z Test results will be stored in test-reports/python-unittest/distributed.test_data_parallel 2023-01-11T22:15:29.1335413Z test_autocast (__main__.TestDataParallel) ... ok (1.858s) 2023-01-11T22:15:29.1335784Z test_data_parallel (__main__.TestDataParallel) ... ok (0.093s) 2023-01-11T22:15:29.1336193Z test_data_parallel_buffers_requiring_grad (__main__.TestDataParallel) ... ok (0.012s) 2023-01-11T22:15:29.1336600Z test_data_parallel_complex (__main__.TestDataParallel) ... ok (0.049s) 2023-01-11T22:15:29.1337014Z test_data_parallel_device_args (__main__.TestDataParallel) ... ok (0.006s) 2023-01-11T22:15:29.1337438Z test_data_parallel_function_deletion (__main__.TestDataParallel) ... ok (0.006s) 2023-01-11T22:15:29.1338323Z test_data_parallel_lazy_linear (__main__.TestDataParallel) ... /opt/conda/lib/python3.10/site-packages/torch/nn/modules/lazy.py:180: UserWarning: Lazy modules are a new feature under heavy development so changes to the API or functionality can happen at any moment. 2023-01-11T22:15:29.1339062Z warnings.warn('Lazy modules are a new feature under heavy development ' 2023-01-11T22:15:29.1339393Z ok (0.002s) 2023-01-11T22:15:29.1339938Z test_data_parallel_model_device (__main__.TestDataParallel) 2023-01-11T22:15:29.1340293Z Test device[0] check at forward time. ... ok (0.033s) 2023-01-11T22:15:29.1340662Z test_data_parallel_model_no_refcycles (__main__.TestDataParallel) ... ok (0.100s) 2023-01-11T22:15:29.1341097Z test_data_parallel_module_zero_inputs (__main__.TestDataParallel) ... ok (0.005s) 2023-01-11T22:15:29.1341974Z test_data_parallel_multiple_input (__main__.TestDataParallel) ... /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/comm.py:231: UserWarning: Using -1 to represent CPU tensor is deprecated. Please use a device object or string instead, e.g., "cpu". 2023-01-11T22:15:29.1342519Z warnings.warn( 2023-01-11T22:15:29.1342755Z ok (0.022s) 2023-01-11T22:15:29.1343074Z test_data_parallel_nested_input (__main__.TestDataParallel) ... ok (0.003s) 2023-01-11T22:15:29.1343789Z test_data_parallel_nested_output (__main__.TestDataParallel) ... ok (0.006s) 2023-01-11T22:15:29.1344536Z test_data_parallel_no_grad (__main__.TestDataParallel) ... ok (0.004s) 2023-01-11T22:15:29.1345245Z test_data_parallel_rnn (__main__.TestDataParallel) ... ok (1.004s) 2023-01-11T22:15:29.1346050Z test_data_parallel_small_back (__main__.TestDataParallel) ... ok (0.003s) 2023-01-11T22:15:29.1346944Z test_data_parallel_sparse (__main__.TestDataParallel) ... ok (0.012s) 2023-01-11T22:15:29.1348061Z test_gather_cpu (__main__.TestDataParallel) ... /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector. 2023-01-11T22:15:29.1348757Z warnings.warn('Was asked to gather along dimension 0, but all ' 2023-01-11T22:15:29.1349062Z ok (0.047s) 2023-01-11T22:15:29.1349370Z test_gather_different_len_dicts (__main__.TestDataParallel) ... ok (0.001s) 2023-01-11T22:15:29.1349763Z test_gather_gpu (__main__.TestDataParallel) ... ok (0.046s) 2023-01-11T22:15:29.1350133Z test_parallel_apply (__main__.TestDataParallel) ... ok (0.005s) 2023-01-11T22:15:29.1350528Z test_parallel_apply_autocast (__main__.TestDataParallel) ... ok (0.006s) 2023-01-11T22:15:29.1350929Z test_parallel_apply_passes_exception (__main__.TestDataParallel) ... ok (0.002s) 2023-01-11T22:15:29.1351494Z test_parameter_list_dict_replica (__main__.TestDataParallel) ... ok (0.007s) 2023-01-11T22:15:29.1351900Z test_replicate (__main__.TestDataParallel) ... ok (0.004s) 2023-01-11T22:15:29.1352257Z test_replicate_buffers (__main__.TestDataParallel) ... ok (0.003s) 2023-01-11T22:15:29.1352645Z test_save_replica_module (__main__.TestDataParallel) ... ok (0.003s) 2023-01-11T22:15:29.1353022Z test_scatter_cpu (__main__.TestDataParallel) ... ok (0.019s) 2023-01-11T22:15:29.1353366Z test_scatter_gpu (__main__.TestDataParallel) ... ok (0.020s) 2023-01-11T22:15:29.1353743Z test_strided_grad_layout (__main__.TestDataParallel) ... ok (1.212s) 2023-01-11T22:15:29.1354115Z test_zero_grad (__main__.TestDataParallel) ... ok (0.008s) 2023-01-11T22:15:29.1354546Z test_data_parallel_module_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.004s) 2023-01-11T22:15:29.1355032Z test_data_parallel_module_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.004s) 2023-01-11T22:15:29.1355528Z test_data_parallel_module_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.004s) 2023-01-11T22:15:29.1356042Z test_data_parallel_module_kwargs_only_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.005s) 2023-01-11T22:15:29.1356559Z test_data_parallel_module_kwargs_only_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.005s) 2023-01-11T22:15:29.1357069Z test_data_parallel_module_kwargs_only_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.005s) 2023-01-11T22:15:29.1357628Z test_data_parallel_module_kwargs_only_empty_dict_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.005s) 2023-01-11T22:15:29.1358187Z test_data_parallel_module_kwargs_only_empty_dict_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.005s) 2023-01-11T22:15:29.1358732Z test_data_parallel_module_kwargs_only_empty_dict_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.005s) 2023-01-11T22:15:29.1359264Z test_data_parallel_module_kwargs_only_empty_list_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.005s) 2023-01-11T22:15:29.1359798Z test_data_parallel_module_kwargs_only_empty_list_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.005s) 2023-01-11T22:15:29.1360340Z test_data_parallel_module_kwargs_only_empty_list_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.005s) 2023-01-11T22:15:29.1360892Z test_data_parallel_module_kwargs_only_empty_tuple_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.005s) 2023-01-11T22:15:29.1361502Z test_data_parallel_module_kwargs_only_empty_tuple_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.005s) 2023-01-11T22:15:29.1362055Z test_data_parallel_module_kwargs_only_empty_tuple_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.005s) 2023-01-11T22:15:29.1362355Z 2023-01-11T22:15:29.1362633Z ---------------------------------------------------------------------- 2023-01-11T22:15:29.1362957Z Ran 46 tests in 4.677s 2023-01-11T22:15:29.1363101Z 2023-01-11T22:15:29.1363197Z OK 2023-01-11T22:15:29.1363329Z 2023-01-11T22:15:29.1363451Z Generating XML reports... 2023-01-11T22:15:29.1364026Z Generated XML report: test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallel-20230111221523.xml 2023-01-11T22:15:29.1364804Z Generated XML report: test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallelDeviceTypeCUDA-20230111221523.xml 2023-01-11T22:15:29.1365191Z 2023-01-11T22:15:29.1365515Z ##[endgroup] 2023-01-11T22:15:29.1366093Z FINISHED PRINTING LOG FILE of distributed/test_data_parallel (/var/lib/jenkins/workspace/test/test-reports/distributed-test_data_parallel_58c257bf) 2023-01-11T22:15:29.1366439Z 2023-01-11T22:15:29.1366695Z Running distributed/fsdp/test_fsdp_traversal ... [2023-01-11 22:15:29.133200] 2023-01-11T22:15:29.1367376Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_traversal.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:15:29.133616] 2023-01-11T22:15:37.1351946Z 2023-01-11T22:15:37.1352454Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_traversal 2023-01-11T22:15:37.1353407Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_traversal (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_traversal_mlxwbzw3) 2023-01-11T22:15:37.1353774Z 2023-01-11T22:15:37.1353890Z Running tests... 2023-01-11T22:15:37.1354371Z ---------------------------------------------------------------------- 2023-01-11T22:15:37.1354942Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal 2023-01-11T22:15:37.1355430Z test_fsdp_modules (__main__.TestTraversal) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:15:37.1355868Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54625 2023-01-11T22:15:37.1356317Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54626 2023-01-11T22:15:37.1356959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:15:37.1357412Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:15:37.1357974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:15:37.1358444Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:15:37.1359018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:15:37.1359468Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:15:37.1360023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:15:37.1360488Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:15:37.1360941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:15:37.1361415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:15:37.1362074Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:15:37.1362756Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:15:37.1363277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:15:37.1363847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:15:37.1364198Z dist init r=1, world=2 2023-01-11T22:15:37.1364451Z dist init r=0, world=2 2023-01-11T22:15:37.1364673Z ok (5.660s) 2023-01-11T22:15:37.1364818Z 2023-01-11T22:15:37.1365089Z ---------------------------------------------------------------------- 2023-01-11T22:15:37.1365417Z Ran 1 test in 5.661s 2023-01-11T22:15:37.1365583Z 2023-01-11T22:15:37.1365676Z OK 2023-01-11T22:15:37.1365792Z 2023-01-11T22:15:37.1365917Z Generating XML reports... 2023-01-11T22:15:37.1366502Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal/TEST-TestTraversal-20230111221531.xml 2023-01-11T22:15:37.1366846Z 2023-01-11T22:15:37.1367146Z ##[endgroup] 2023-01-11T22:15:37.1367733Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_traversal (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_traversal_mlxwbzw3) 2023-01-11T22:15:37.1368102Z 2023-01-11T22:15:37.1368373Z Running distributed/fsdp/test_fsdp_uneven ... [2023-01-11 22:15:37.135189] 2023-01-11T22:15:37.1369059Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_uneven.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:15:37.135450] 2023-01-11T22:15:45.5407435Z 2023-01-11T22:15:45.5408812Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_uneven 2023-01-11T22:15:45.5410694Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_uneven (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_uneven_0905syut) 2023-01-11T22:15:45.5411451Z 2023-01-11T22:15:45.5411689Z Running tests... 2023-01-11T22:15:45.5412519Z ---------------------------------------------------------------------- 2023-01-11T22:15:45.5413161Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_uneven 2023-01-11T22:15:45.5414052Z test_one_iteration (__main__.TestUnevenParamShard) 2023-01-11T22:15:45.5414837Z Test FSDP with uneven divide of parameter shards. ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:15:45.5415311Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54739 2023-01-11T22:15:45.5415741Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54740 2023-01-11T22:15:45.5416383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:15:45.5416841Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:15:45.5417420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:15:45.5417873Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:15:45.5418448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:15:45.5418892Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:15:45.5419443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:15:45.5419911Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:15:45.5420361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:15:45.5420862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:15:45.5421499Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:15:45.5422183Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:15:45.5422703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:15:45.5423341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:15:45.5423682Z dist init r=1, world=2 2023-01-11T22:15:45.5423934Z dist init r=0, world=2 2023-01-11T22:15:45.5424175Z ok (6.048s) 2023-01-11T22:15:45.5424325Z 2023-01-11T22:15:45.5424576Z ---------------------------------------------------------------------- 2023-01-11T22:15:45.5424900Z Ran 1 test in 6.048s 2023-01-11T22:15:45.5425062Z 2023-01-11T22:15:45.5425159Z OK 2023-01-11T22:15:45.5425297Z 2023-01-11T22:15:45.5425404Z Generating XML reports... 2023-01-11T22:15:45.5426013Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_uneven/TEST-TestUnevenParamShard-20230111221539.xml 2023-01-11T22:15:45.5426382Z 2023-01-11T22:15:45.5426703Z ##[endgroup] 2023-01-11T22:15:45.5427273Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_uneven (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_uneven_0905syut) 2023-01-11T22:15:45.5427635Z 2023-01-11T22:15:45.5427950Z Running distributed/_shard/sharded_tensor/ops/test_embedding_bag ... [2023-01-11 22:15:45.540773] 2023-01-11T22:15:45.5428703Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_embedding_bag.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:15:45.541070] 2023-01-11T22:15:54.3022237Z 2023-01-11T22:15:54.3023189Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_embedding_bag 2023-01-11T22:15:54.3024483Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_embedding_bag (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_embedding_bag_iz811xwu) 2023-01-11T22:15:54.3025015Z 2023-01-11T22:15:54.3025133Z Running tests... 2023-01-11T22:15:54.3025698Z ---------------------------------------------------------------------- 2023-01-11T22:15:54.3026509Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding_bag 2023-01-11T22:15:54.3027226Z test_sharded_embedding_bag_colwise (__main__.TestShardedEmbeddingBag) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:15:54.3027847Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54857 2023-01-11T22:15:54.3028279Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54858 2023-01-11T22:15:54.3028972Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 54859 2023-01-11T22:15:54.3029418Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 54860 2023-01-11T22:15:54.3030297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:15:54.3030740Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:15:54.3031573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:15:54.3032057Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:15:54.3032875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:15:54.3033334Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:15:54.3034190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:15:54.3034671Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:15:54.3035474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:15:54.3035930Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:15:54.3036685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:15:54.3037352Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:15:54.3038165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:15:54.3038630Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:15:54.3039394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:15:54.3039915Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:15:54.3040334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T22:15:54.3041052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T22:15:54.3041520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:15:54.3042207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:15:54.3042605Z skip: Need at least 4 CUDA devices (4.055s) 2023-01-11T22:15:54.3043194Z test_sharded_embedding_bag_rowwise (__main__.TestShardedEmbeddingBag) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54993 2023-01-11T22:15:54.3043894Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54994 2023-01-11T22:15:54.3044321Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 54995 2023-01-11T22:15:54.3045074Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 54996 2023-01-11T22:15:54.3045726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:15:54.3046392Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:15:54.3046969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:15:54.3047692Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:15:54.3048311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:15:54.3048986Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:15:54.3049562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:15:54.3050489Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:15:54.3051089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:15:54.3051791Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:15:54.3052368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:15:54.3053081Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:15:54.3053648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:15:54.3054344Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:15:54.3054924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:15:54.3055607Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:15:54.3056050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T22:15:54.3056558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:15:54.3057236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T22:15:54.3057682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:15:54.3058446Z skip: Need at least 4 CUDA devices (2.410s) 2023-01-11T22:15:54.3058643Z 2023-01-11T22:15:54.3058925Z ---------------------------------------------------------------------- 2023-01-11T22:15:54.3059311Z Ran 2 tests in 6.465s 2023-01-11T22:15:54.3059590Z 2023-01-11T22:15:54.3059741Z OK (skipped=2) 2023-01-11T22:15:54.3059898Z 2023-01-11T22:15:54.3060023Z Generating XML reports... 2023-01-11T22:15:54.3060840Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding_bag/TEST-TestShardedEmbeddingBag-20230111221547.xml 2023-01-11T22:15:54.3061324Z 2023-01-11T22:15:54.3061628Z ##[endgroup] 2023-01-11T22:15:54.3062575Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_embedding_bag (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_embedding_bag_iz811xwu) 2023-01-11T22:15:54.3063001Z 2023-01-11T22:15:54.3063387Z Running distributed/_shard/sharded_optim/test_sharded_optim ... [2023-01-11 22:15:54.302269] 2023-01-11T22:15:54.3064289Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_optim/test_sharded_optim.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:15:54.302529] 2023-01-11T22:16:02.9054302Z 2023-01-11T22:16:02.9054818Z Expand the folded group to see the log file of distributed/_shard/sharded_optim/test_sharded_optim 2023-01-11T22:16:02.9056271Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_optim/test_sharded_optim (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_optim-test_sharded_optim_w9zyvcei) 2023-01-11T22:16:02.9056714Z 2023-01-11T22:16:02.9056828Z Running tests... 2023-01-11T22:16:02.9057314Z ---------------------------------------------------------------------- 2023-01-11T22:16:02.9058123Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_optim.test_sharded_optim 2023-01-11T22:16:02.9059108Z test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:16:02.9060009Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55164 2023-01-11T22:16:02.9060930Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55165 2023-01-11T22:16:02.9061894Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 55166 2023-01-11T22:16:02.9062851Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 55167 2023-01-11T22:16:02.9064138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:02.9064603Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:02.9065208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:02.9065680Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:02.9066260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:02.9066709Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:02.9067256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:02.9067722Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:02.9068297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:02.9068721Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:02.9069286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:02.9069749Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:02.9070319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:02.9070900Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:02.9071468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:02.9071929Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:02.9072386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:16:02.9072822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T22:16:02.9073280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T22:16:02.9073745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:16:02.9074119Z skip: Need at least 4 CUDA devices (3.908s) 2023-01-11T22:16:02.9074586Z test_sharded_optim (__main__.TestShardedOptimizer) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55300 2023-01-11T22:16:02.9075118Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55301 2023-01-11T22:16:02.9075560Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 55302 2023-01-11T22:16:02.9076005Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 55303 2023-01-11T22:16:02.9076640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:02.9077100Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:02.9077654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:02.9078125Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:02.9078692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:02.9079139Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:02.9079693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:02.9080151Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:02.9080720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:02.9081146Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:02.9081707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:02.9082187Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:02.9082743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:02.9083166Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:02.9083737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:02.9084194Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:02.9084643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T22:16:02.9085085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:16:02.9085544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T22:16:02.9086007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:16:02.9086380Z skip: Need at least 4 CUDA devices (2.411s) 2023-01-11T22:16:02.9086572Z 2023-01-11T22:16:02.9086845Z ---------------------------------------------------------------------- 2023-01-11T22:16:02.9087201Z Ran 2 tests in 6.319s 2023-01-11T22:16:02.9087400Z 2023-01-11T22:16:02.9087507Z OK (skipped=2) 2023-01-11T22:16:02.9087643Z 2023-01-11T22:16:02.9087765Z Generating XML reports... 2023-01-11T22:16:02.9088490Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_optim.test_sharded_optim/TEST-TestShardedOptimizer-20230111221556.xml 2023-01-11T22:16:02.9088935Z 2023-01-11T22:16:02.9089259Z ##[endgroup] 2023-01-11T22:16:02.9089916Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_optim/test_sharded_optim (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_optim-test_sharded_optim_w9zyvcei) 2023-01-11T22:16:02.9090700Z 2023-01-11T22:16:02.9091034Z Running distributed/_shard/sharded_tensor/ops/test_init ... [2023-01-11 22:16:02.905304] 2023-01-11T22:16:02.9091748Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_init.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:16:02.905564] 2023-01-11T22:16:13.9834806Z 2023-01-11T22:16:13.9835332Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_init 2023-01-11T22:16:13.9836285Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_init (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_init_jti4_c5i) 2023-01-11T22:16:13.9836688Z 2023-01-11T22:16:13.9836788Z Running tests... 2023-01-11T22:16:13.9837286Z ---------------------------------------------------------------------- 2023-01-11T22:16:13.9838246Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_init 2023-01-11T22:16:13.9838868Z test_init_sharded_tensor_with_kaiming_uniform (__main__.TestShardedTensorNNInit) 2023-01-11T22:16:13.9839425Z Test torch.nn.init.kaiming_uniform_(ShardedTensor, a, mode, nonlinearit) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:16:13.9839908Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55471 2023-01-11T22:16:13.9840362Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55472 2023-01-11T22:16:13.9840804Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 55473 2023-01-11T22:16:13.9841225Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 55474 2023-01-11T22:16:13.9841855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:13.9842311Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:13.9842864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:13.9843337Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:13.9843911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:13.9844354Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:13.9844908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:13.9845374Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:13.9845944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:13.9846369Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:13.9846939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:13.9847399Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:13.9847969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:13.9848389Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:13.9849155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:13.9849618Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:13.9850049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:16:13.9850697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:16:13.9851165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T22:16:13.9851628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T22:16:13.9852000Z skip: Need at least 4 CUDA devices (3.956s) 2023-01-11T22:16:13.9852375Z test_init_sharded_tensor_with_normal (__main__.TestShardedTensorNNInit) 2023-01-11T22:16:13.9852896Z Test torch.nn.init.normal_(ShardedTensor, mean, std) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55607 2023-01-11T22:16:13.9853418Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55608 2023-01-11T22:16:13.9853844Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 55609 2023-01-11T22:16:13.9854275Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 55610 2023-01-11T22:16:13.9854888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:13.9855410Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:13.9855998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:13.9856467Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:13.9857037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:13.9857465Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:13.9858028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:13.9858490Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:13.9859059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:13.9859484Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:13.9860051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:13.9860509Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:13.9861059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:13.9861504Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:13.9862071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:13.9862531Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:13.9862944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T22:16:13.9863408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:16:13.9863878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:16:13.9864326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T22:16:13.9864713Z skip: Need at least 4 CUDA devices (2.410s) 2023-01-11T22:16:13.9865092Z test_init_sharded_tensor_with_uniform (__main__.TestShardedTensorNNInit) 2023-01-11T22:16:13.9865611Z Test torch.nn.init.uniform_(ShardedTensor, a, b) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55743 2023-01-11T22:16:13.9866199Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55744 2023-01-11T22:16:13.9866646Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 55745 2023-01-11T22:16:13.9867085Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 55746 2023-01-11T22:16:13.9867680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:13.9868127Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:13.9868695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:13.9869159Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:13.9869711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:13.9870163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:13.9870729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:13.9871189Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:13.9871742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:13.9872239Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:13.9872814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:13.9873257Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:13.9873829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:13.9874269Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:13.9874837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:13.9875280Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:13.9875710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T22:16:13.9876180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:16:13.9876625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:16:13.9877085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T22:16:13.9877469Z skip: Need at least 4 CUDA devices (2.411s) 2023-01-11T22:16:13.9877663Z 2023-01-11T22:16:13.9877932Z ---------------------------------------------------------------------- 2023-01-11T22:16:13.9878242Z Ran 3 tests in 8.777s 2023-01-11T22:16:13.9878407Z 2023-01-11T22:16:13.9878512Z OK (skipped=3) 2023-01-11T22:16:13.9878668Z 2023-01-11T22:16:13.9878790Z Generating XML reports... 2023-01-11T22:16:13.9879405Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_init/TEST-TestShardedTensorNNInit-20230111221604.xml 2023-01-11T22:16:13.9879793Z 2023-01-11T22:16:13.9880110Z ##[endgroup] 2023-01-11T22:16:13.9880751Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_init (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_init_jti4_c5i) 2023-01-11T22:16:13.9881139Z 2023-01-11T22:16:13.9881443Z Running distributed/_shard/sharding_spec/test_sharding_spec ... [2023-01-11 22:16:13.983554] 2023-01-11T22:16:13.9882173Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharding_spec/test_sharding_spec.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:16:13.983832] 2023-01-11T22:16:25.0966399Z 2023-01-11T22:16:25.0967219Z Expand the folded group to see the log file of distributed/_shard/sharding_spec/test_sharding_spec 2023-01-11T22:16:25.0968488Z ##[group]PRINTING LOG FILE of distributed/_shard/sharding_spec/test_sharding_spec (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharding_spec-test_sharding_spec_e_en34zw) 2023-01-11T22:16:25.0969425Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6dezfz8t 2023-01-11T22:16:25.0969941Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6dezfz8t/_remote_module_non_scriptable.py 2023-01-11T22:16:25.0970255Z 2023-01-11T22:16:25.0970645Z Running tests... 2023-01-11T22:16:25.0971171Z ---------------------------------------------------------------------- 2023-01-11T22:16:25.0971750Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec 2023-01-11T22:16:25.0972286Z test_custom_sharding_spec (__main__.TestCustomShardingSpec) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:16:25.0972798Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55914 2023-01-11T22:16:25.0973271Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55915 2023-01-11T22:16:25.0973707Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 55916 2023-01-11T22:16:25.0974101Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 55917 2023-01-11T22:16:25.0974848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:25.0975322Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:25.0975904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:25.0976373Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:25.0976950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:25.0977395Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:25.0977912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:25.0978394Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:25.0978994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:25.0979438Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:25.0979961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:25.0980427Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:25.0980995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:25.0981469Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:25.0982017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:25.0982485Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:25.0982934Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_1il8z66 2023-01-11T22:16:25.0983427Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_1il8z66/_remote_module_non_scriptable.py 2023-01-11T22:16:25.0983962Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpilf0lezg 2023-01-11T22:16:25.0984486Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpilf0lezg/_remote_module_non_scriptable.py 2023-01-11T22:16:25.0984990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:16:25.0985440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T22:16:25.0986015Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplcmw5x2t 2023-01-11T22:16:25.0986546Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplcmw5x2t/_remote_module_non_scriptable.py 2023-01-11T22:16:25.0987056Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpabt9mv4s 2023-01-11T22:16:25.0987588Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpabt9mv4s/_remote_module_non_scriptable.py 2023-01-11T22:16:25.0988115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T22:16:25.0988593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:16:25.0988938Z ok (3.923s) 2023-01-11T22:16:25.0989256Z test_custom_sharding_spec_shard_tensor (__main__.TestCustomShardingSpec) 2023-01-11T22:16:25.0989767Z Test custom spec can be invoked from the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56050 2023-01-11T22:16:25.0990260Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56051 2023-01-11T22:16:25.0990686Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 56052 2023-01-11T22:16:25.0991123Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 56053 2023-01-11T22:16:25.0991792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:25.0992278Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:25.0992805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:25.0993270Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:25.0993850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:25.0994287Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:25.0994837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:25.0995312Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:25.0995869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:25.0996289Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:25.0996856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:25.0997314Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:25.0997894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:25.0998304Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:25.0998865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:25.0999322Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:25.0999761Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8fwknn8a 2023-01-11T22:16:25.1000298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8fwknn8a/_remote_module_non_scriptable.py 2023-01-11T22:16:25.1000824Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpel4yu4u5 2023-01-11T22:16:25.1001353Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpel4yu4u5/_remote_module_non_scriptable.py 2023-01-11T22:16:25.1001853Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvaidqsbj 2023-01-11T22:16:25.1002378Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvaidqsbj/_remote_module_non_scriptable.py 2023-01-11T22:16:25.1002950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:16:25.1003421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T22:16:25.1003864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:16:25.1004353Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbx7x66q7 2023-01-11T22:16:25.1004884Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbx7x66q7/_remote_module_non_scriptable.py 2023-01-11T22:16:25.1005363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T22:16:25.1005752Z skip: Need at least 4 CUDA devices (2.410s) 2023-01-11T22:16:25.1006147Z test_custom_sharding_spec_tensor_ctor (__main__.TestCustomShardingSpec) 2023-01-11T22:16:25.1006634Z Test sharded_tensor.ones(...) with the custom ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56186 2023-01-11T22:16:25.1007125Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56187 2023-01-11T22:16:25.1007587Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 56188 2023-01-11T22:16:25.1008001Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 56189 2023-01-11T22:16:25.1008649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:25.1009107Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:25.1009680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:25.1010147Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:25.1010929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:25.1011384Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:25.1011954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:25.1012396Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:25.1012969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:25.1013412Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:25.1013974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:25.1014418Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:25.1014990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:25.1015432Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:25.1016023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:25.1016437Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:25.1016897Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3uq0_ip3 2023-01-11T22:16:25.1017434Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3uq0_ip3/_remote_module_non_scriptable.py 2023-01-11T22:16:25.1017943Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx48wcer8 2023-01-11T22:16:25.1018472Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx48wcer8/_remote_module_non_scriptable.py 2023-01-11T22:16:25.1018993Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdgx5zf3t 2023-01-11T22:16:25.1030765Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsc8qna00 2023-01-11T22:16:25.1031528Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdgx5zf3t/_remote_module_non_scriptable.py 2023-01-11T22:16:25.1032078Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsc8qna00/_remote_module_non_scriptable.py 2023-01-11T22:16:25.1032568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T22:16:25.1033056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T22:16:25.1033500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:16:25.1033966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:16:25.1034340Z skip: Need at least 4 CUDA devices (2.410s) 2023-01-11T22:16:25.1034700Z test_check_overlapping (__main__.TestShardingSpec) ... ok (0.002s) 2023-01-11T22:16:25.1035099Z test_chunked_sharding_spec (__main__.TestShardingSpec) ... ok (0.011s) 2023-01-11T22:16:25.1035480Z test_device_placement (__main__.TestShardingSpec) ... ok (0.006s) 2023-01-11T22:16:25.1035873Z test_enumerable_sharding_spec (__main__.TestShardingSpec) ... ok (0.007s) 2023-01-11T22:16:25.1036287Z test_get_chunk_sharding_params (__main__.TestShardingSpec) ... ok (0.002s) 2023-01-11T22:16:25.1036689Z test_get_chunked_dim_size (__main__.TestShardingSpec) ... ok (0.001s) 2023-01-11T22:16:25.1037132Z test_get_split_size (__main__.TestShardingSpec) ... ok (0.001s) 2023-01-11T22:16:25.1037555Z test_infer_sharding_spec_from_shards_metadata (__main__.TestShardingSpec) ... ok (0.010s) 2023-01-11T22:16:25.1037813Z 2023-01-11T22:16:25.1038117Z ---------------------------------------------------------------------- 2023-01-11T22:16:25.1038458Z Ran 11 tests in 8.783s 2023-01-11T22:16:25.1038596Z 2023-01-11T22:16:25.1038711Z OK (skipped=2) 2023-01-11T22:16:25.1038860Z 2023-01-11T22:16:25.1038982Z Generating XML reports... 2023-01-11T22:16:25.1039632Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestCustomShardingSpec-20230111221615.xml 2023-01-11T22:16:25.1040445Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestShardingSpec-20230111221615.xml 2023-01-11T22:16:25.1040809Z 2023-01-11T22:16:25.1041145Z ##[endgroup] 2023-01-11T22:16:25.1041811Z FINISHED PRINTING LOG FILE of distributed/_shard/sharding_spec/test_sharding_spec (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharding_spec-test_sharding_spec_e_en34zw) 2023-01-11T22:16:25.1042215Z 2023-01-11T22:16:25.1042466Z Running distributed/fsdp/test_fsdp_input ... [2023-01-11 22:16:25.096788] 2023-01-11T22:16:25.1043138Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_input.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:16:25.097119] 2023-01-11T22:16:37.2694484Z 2023-01-11T22:16:37.2695366Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_input 2023-01-11T22:16:37.2696454Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_input (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_input_g6h0fi32) 2023-01-11T22:16:37.2696805Z 2023-01-11T22:16:37.2696918Z Running tests... 2023-01-11T22:16:37.2697450Z ---------------------------------------------------------------------- 2023-01-11T22:16:37.2698002Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_input 2023-01-11T22:16:37.2698407Z test_input_type_dict (__main__.TestInput) 2023-01-11T22:16:37.2698820Z Test FSDP with input being a list or a dict, only single GPU. ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:16:37.2699280Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56357 2023-01-11T22:16:37.2699904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:37.2700604Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:37.2701179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:37.2701634Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:37.2702081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:16:37.2702742Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2023-01-11T22:16:37.2703272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:16:37.2704022Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:288: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1. 2023-01-11T22:16:37.2704513Z warnings.warn( 2023-01-11T22:16:37.2705671Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:16:37.2706552Z warnings.warn( 2023-01-11T22:16:37.2706797Z dist init r=0, world=1 2023-01-11T22:16:37.2707029Z ok (5.732s) 2023-01-11T22:16:37.2707298Z test_input_type_list (__main__.TestInput) 2023-01-11T22:16:37.2707763Z Test FSDP with input being a list or a dict, only single GPU. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56399 2023-01-11T22:16:37.2708429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:37.2708880Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:37.2709451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:37.2709898Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:37.2710343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:16:37.2711000Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2023-01-11T22:16:37.2711515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:16:37.2712266Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:288: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1. 2023-01-11T22:16:37.2712750Z warnings.warn( 2023-01-11T22:16:37.2713893Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:16:37.2714668Z warnings.warn( 2023-01-11T22:16:37.2714897Z dist init r=0, world=1 2023-01-11T22:16:37.2715131Z ok (4.110s) 2023-01-11T22:16:37.2715276Z 2023-01-11T22:16:37.2715539Z ---------------------------------------------------------------------- 2023-01-11T22:16:37.2715863Z Ran 2 tests in 9.842s 2023-01-11T22:16:37.2716010Z 2023-01-11T22:16:37.2716101Z OK 2023-01-11T22:16:37.2716232Z 2023-01-11T22:16:37.2716353Z Generating XML reports... 2023-01-11T22:16:37.2716907Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_input/TEST-TestInput-20230111221627.xml 2023-01-11T22:16:37.2717300Z 2023-01-11T22:16:37.2717582Z ##[endgroup] 2023-01-11T22:16:37.2718167Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_input (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_input_g6h0fi32) 2023-01-11T22:16:37.2718517Z 2023-01-11T22:16:37.2718812Z Running distributed/tensor/parallel/test_2d_parallel ... [2023-01-11 22:16:37.269513] 2023-01-11T22:16:37.2719523Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/tensor/parallel/test_2d_parallel.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:16:37.269780] 2023-01-11T22:16:52.9631600Z 2023-01-11T22:16:52.9632351Z Expand the folded group to see the log file of distributed/tensor/parallel/test_2d_parallel 2023-01-11T22:16:52.9633452Z ##[group]PRINTING LOG FILE of distributed/tensor/parallel/test_2d_parallel (/var/lib/jenkins/workspace/test/test-reports/distributed-tensor-parallel-test_2d_parallel_bnm0aezv) 2023-01-11T22:16:52.9633868Z 2023-01-11T22:16:52.9633985Z Running tests... 2023-01-11T22:16:52.9634479Z ---------------------------------------------------------------------- 2023-01-11T22:16:52.9635081Z Test results will be stored in test-reports/python-unittest/distributed.tensor.parallel.test_2d_parallel 2023-01-11T22:16:52.9635650Z test_2d_fsdp_integration_correctness (__main__.Test2dParallelIntegration) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:16:52.9636397Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56476 2023-01-11T22:16:52.9636868Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56477 2023-01-11T22:16:52.9637481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:52.9637939Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:52.9638494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:52.9638968Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:52.9639585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:52.9640029Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:52.9640624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:52.9641098Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:52.9641531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:16:52.9642274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:16:52.9643120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:16:52.9644179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:16:52.9645137Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:16:52.9645831Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:16:52.9646252Z skip: Need at least 4 CUDA devices (3.896s) 2023-01-11T22:16:52.9646775Z test_2d_fsdp_integration_fsdp_nested (__main__.Test2dParallelIntegration) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56546 2023-01-11T22:16:52.9647338Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56547 2023-01-11T22:16:52.9647913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:52.9648366Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:52.9649094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:52.9649558Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:52.9650115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:52.9651099Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:52.9651684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:52.9652149Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:52.9652566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:16:52.9653056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:16:52.9653544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:16:52.9654001Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:16:52.9654655Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:16:52.9655436Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:16:52.9655887Z skip: Need at least 4 CUDA devices (2.407s) 2023-01-11T22:16:52.9656387Z test_2d_fsdp_integration_fsdp_nested_param_groups (__main__.Test2dParallelIntegration) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56616 2023-01-11T22:16:52.9656953Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56617 2023-01-11T22:16:52.9657562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:52.9658015Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:52.9658567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:52.9659033Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:52.9659603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:52.9660030Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:52.9660596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:52.9661055Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:52.9661486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:16:52.9661953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:16:52.9662432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:16:52.9662915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:16:52.9663543Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:16:52.9664228Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:16:52.9664666Z skip: Need at least 4 CUDA devices (2.407s) 2023-01-11T22:16:52.9665170Z test_2d_fsdp_integration_functionality (__main__.Test2dParallelIntegration) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56686 2023-01-11T22:16:52.9665710Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56687 2023-01-11T22:16:52.9666421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:52.9666866Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:52.9667428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:52.9667877Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:52.9668449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:52.9668890Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:52.9669439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:52.9669898Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:52.9670330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:16:52.9670816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:16:52.9671277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:16:52.9671755Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:16:52.9672497Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:16:52.9673189Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:16:52.9673610Z skip: Need at least 4 CUDA devices (2.308s) 2023-01-11T22:16:52.9674115Z test_2d_fsdp_integration_use_orig_params (__main__.Test2dParallelIntegration) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56756 2023-01-11T22:16:52.9674676Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56757 2023-01-11T22:16:52.9675280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:52.9675708Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:52.9676277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:52.9676743Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:52.9677314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:16:52.9677738Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:16:52.9678301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:16:52.9678758Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:16:52.9679176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:16:52.9679661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:16:52.9680138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:16:52.9680623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:16:52.9681252Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:16:52.9681929Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:16:52.9682367Z skip: Need at least 4 CUDA devices (2.307s) 2023-01-11T22:16:52.9682559Z 2023-01-11T22:16:52.9682827Z ---------------------------------------------------------------------- 2023-01-11T22:16:52.9683204Z Ran 5 tests in 13.326s 2023-01-11T22:16:52.9683366Z 2023-01-11T22:16:52.9683474Z OK (skipped=5) 2023-01-11T22:16:52.9683625Z 2023-01-11T22:16:52.9683747Z Generating XML reports... 2023-01-11T22:16:52.9684386Z Generated XML report: test-reports/python-unittest/distributed.tensor.parallel.test_2d_parallel/TEST-Test2dParallelIntegration-20230111221639.xml 2023-01-11T22:16:52.9684787Z 2023-01-11T22:16:52.9685117Z ##[endgroup] 2023-01-11T22:16:52.9685772Z FINISHED PRINTING LOG FILE of distributed/tensor/parallel/test_2d_parallel (/var/lib/jenkins/workspace/test/test-reports/distributed-tensor-parallel-test_2d_parallel_bnm0aezv) 2023-01-11T22:16:52.9686171Z 2023-01-11T22:16:52.9686417Z Running distributed/_tensor/test_math_ops ... [2023-01-11 22:16:52.963315] 2023-01-11T22:16:52.9687082Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_math_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:16:52.963591] 2023-01-11T22:17:08.8441910Z 2023-01-11T22:17:08.8442694Z Expand the folded group to see the log file of distributed/_tensor/test_math_ops 2023-01-11T22:17:08.8443649Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_math_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_math_ops_klbb1ug9) 2023-01-11T22:17:08.8443972Z 2023-01-11T22:17:08.8444092Z Running tests... 2023-01-11T22:17:08.8444625Z ---------------------------------------------------------------------- 2023-01-11T22:17:08.8445436Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_math_ops 2023-01-11T22:17:08.8445978Z test_softmax_fwd (__main__.DistMathOpsTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:17:08.8446442Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56861 2023-01-11T22:17:08.8446908Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56862 2023-01-11T22:17:08.8447551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:08.8448024Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:08.8448611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:08.8449362Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:08.8450550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:08.8451031Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:08.8451651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:08.8452142Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:08.8452577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:17:08.8453075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:17:08.8453576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:17:08.8454092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:17:08.8454757Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:17:08.8455475Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:17:08.8455901Z ok (5.542s) 2023-01-11T22:17:08.8456315Z test_softmax_with_bwd (__main__.DistMathOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56942 2023-01-11T22:17:08.8456840Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56943 2023-01-11T22:17:08.8457623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:08.8458095Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:08.8458709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:08.8459184Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:08.8459779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:08.8460251Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:08.8460814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:08.8461297Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:08.8461747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:17:08.8462261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:17:08.8462751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:17:08.8463255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:17:08.8464025Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:17:08.8464754Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:17:08.8465160Z ok (4.111s) 2023-01-11T22:17:08.8465558Z test_sum (__main__.DistMathOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57027 2023-01-11T22:17:08.8466079Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57028 2023-01-11T22:17:08.8466690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:08.8467160Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:08.8467739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:08.8468229Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:08.8468803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:08.8469269Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:08.8469851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:08.8470338Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:08.8470771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:17:08.8471274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:17:08.8471780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:17:08.8472261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:17:08.8472940Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:17:08.8473654Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:17:08.8474076Z ok (3.910s) 2023-01-11T22:17:08.8474213Z 2023-01-11T22:17:08.8474478Z ---------------------------------------------------------------------- 2023-01-11T22:17:08.8474810Z Ran 3 tests in 13.564s 2023-01-11T22:17:08.8475046Z 2023-01-11T22:17:08.8475138Z OK 2023-01-11T22:17:08.8475275Z 2023-01-11T22:17:08.8475380Z Generating XML reports... 2023-01-11T22:17:08.8475969Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_math_ops/TEST-DistMathOpsTest-20230111221654.xml 2023-01-11T22:17:08.8476352Z 2023-01-11T22:17:08.8476676Z ##[endgroup] 2023-01-11T22:17:08.8477244Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_math_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_math_ops_klbb1ug9) 2023-01-11T22:17:08.8477627Z 2023-01-11T22:17:08.8477938Z Running distributed/_shard/sharded_tensor/ops/test_tensor_ops ... [2023-01-11 22:17:08.844276] 2023-01-11T22:17:08.8478681Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_tensor_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:17:08.844525] 2023-01-11T22:17:24.4702753Z 2023-01-11T22:17:24.4703548Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_tensor_ops 2023-01-11T22:17:24.4704622Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_tensor_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_tensor_ops_cna5kufx) 2023-01-11T22:17:24.4705002Z 2023-01-11T22:17:24.4705120Z Running tests... 2023-01-11T22:17:24.4705656Z ---------------------------------------------------------------------- 2023-01-11T22:17:24.4706491Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_tensor_ops 2023-01-11T22:17:24.4707037Z test_clone (__main__.TestTensorOps) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:17:24.4707495Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57143 2023-01-11T22:17:24.4707967Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57144 2023-01-11T22:17:24.4708424Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 57145 2023-01-11T22:17:24.4708890Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 57146 2023-01-11T22:17:24.4709523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:24.4709979Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:24.4710555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:24.4711078Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:24.4711626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:24.4712093Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:24.4712671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:24.4713147Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:24.4713733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:24.4714195Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:24.4714772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:24.4715250Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:24.4715837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:24.4716299Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:24.4716858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:24.4717341Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:24.4717896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:17:24.4718386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T22:17:24.4718855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:17:24.4719332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T22:17:24.4719802Z skip: Need at least 4 CUDA devices (3.920s) 2023-01-11T22:17:24.4720257Z test_deep_copy (__main__.TestTensorOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57279 2023-01-11T22:17:24.4720767Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57280 2023-01-11T22:17:24.4721232Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 57281 2023-01-11T22:17:24.4721689Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 57282 2023-01-11T22:17:24.4722322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:24.4722776Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:24.4723354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:24.4723843Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:24.4724500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:24.4724956Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:24.4725535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:24.4726020Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:24.4726593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:24.4727056Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:24.4727629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:24.4728109Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:24.4728677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:24.4729143Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:24.4729715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:24.4730181Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:24.4730950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T22:17:24.4731451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T22:17:24.4731932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:17:24.4732398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:17:24.4732802Z skip: Need at least 4 CUDA devices (2.309s) 2023-01-11T22:17:24.4733252Z test_detach (__main__.TestTensorOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57415 2023-01-11T22:17:24.4733775Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57416 2023-01-11T22:17:24.4734220Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 57417 2023-01-11T22:17:24.4734671Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 57418 2023-01-11T22:17:24.4735300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:24.4735856Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:24.4736437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:24.4736924Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:24.4737508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:24.4737958Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:24.4738533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:24.4739015Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:24.4739582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:24.4740045Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:24.4740619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:24.4741098Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:24.4741742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:24.4742220Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:24.4742797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:24.4743264Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:24.4743707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T22:17:24.4744201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T22:17:24.4744686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:17:24.4745151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:17:24.4745553Z skip: Need at least 4 CUDA devices (2.409s) 2023-01-11T22:17:24.4746014Z test_inplace_copy (__main__.TestTensorOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57551 2023-01-11T22:17:24.4746546Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57552 2023-01-11T22:17:24.4746992Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 57553 2023-01-11T22:17:24.4747441Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 57554 2023-01-11T22:17:24.4748065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:24.4748519Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:24.4749095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:24.4749579Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:24.4750158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:24.4750603Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:24.4751173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:24.4751652Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:24.4752264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:24.4752794Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:24.4753367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:24.4753850Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:24.4754416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:24.4754888Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:24.4755465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:24.4755947Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:24.4756373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:17:24.4756860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T22:17:24.4757346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T22:17:24.4757812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:17:24.4758212Z skip: Need at least 4 CUDA devices (2.309s) 2023-01-11T22:17:24.4758672Z test_set_requires_grad (__main__.TestTensorOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57687 2023-01-11T22:17:24.4759262Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57688 2023-01-11T22:17:24.4759715Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 57689 2023-01-11T22:17:24.4760169Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 57690 2023-01-11T22:17:24.4760789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:24.4761242Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:24.4761823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:24.4762314Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:24.4762899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:24.4763349Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:24.4763923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:24.4764407Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:24.4764973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:24.4765431Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:24.4766005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:24.4766485Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:24.4767052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:24.4767517Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:24.4768088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:24.4768566Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:24.4768991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:17:24.4769481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T22:17:24.4769965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T22:17:24.4770765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:17:24.4771176Z skip: Need at least 4 CUDA devices (2.409s) 2023-01-11T22:17:24.4771378Z 2023-01-11T22:17:24.4771652Z ---------------------------------------------------------------------- 2023-01-11T22:17:24.4771984Z Ran 5 tests in 13.358s 2023-01-11T22:17:24.4772137Z 2023-01-11T22:17:24.4772243Z OK (skipped=5) 2023-01-11T22:17:24.4772404Z 2023-01-11T22:17:24.4772526Z Generating XML reports... 2023-01-11T22:17:24.4773147Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_tensor_ops/TEST-TestTensorOps-20230111221710.xml 2023-01-11T22:17:24.4773556Z 2023-01-11T22:17:24.4773859Z ##[endgroup] 2023-01-11T22:17:24.4774526Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_tensor_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_tensor_ops_cna5kufx) 2023-01-11T22:17:24.4774977Z 2023-01-11T22:17:24.4775239Z Running distributed/fsdp/test_fsdp_apply ... [2023-01-11 22:17:24.470332] 2023-01-11T22:17:24.4775921Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_apply.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:17:24.470581] 2023-01-11T22:17:40.8963075Z 2023-01-11T22:17:40.8963548Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_apply 2023-01-11T22:17:40.8964850Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_apply (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_apply_0b_ajz4z) 2023-01-11T22:17:40.8965233Z 2023-01-11T22:17:40.8965351Z Running tests... 2023-01-11T22:17:40.8965885Z ---------------------------------------------------------------------- 2023-01-11T22:17:40.8966695Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_apply 2023-01-11T22:17:40.8967180Z test_apply_in_summon_raises_error (__main__.TestApply) 2023-01-11T22:17:40.8967633Z Tests that calling ``apply()`` on an FSDP instance inside the ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:17:40.8968240Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57858 2023-01-11T22:17:40.8968711Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57859 2023-01-11T22:17:40.8969564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:40.8970050Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:40.8971140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:40.8971644Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:40.8972227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:40.8972713Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:40.8973299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:40.8973773Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:40.8974237Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:17:40.8974761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:17:40.8975446Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:17:40.8976149Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:17:40.8976700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:17:40.8977355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:17:40.8978652Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:17:40.8979546Z warnings.warn( 2023-01-11T22:17:40.8980687Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:17:40.8981548Z warnings.warn( 2023-01-11T22:17:40.8981825Z File "", line 1, in 2023-01-11T22:17:40.8982202Z File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main 2023-01-11T22:17:40.8982594Z exitcode = _main(fd, parent_sentinel) 2023-01-11T22:17:40.8983074Z File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main 2023-01-11T22:17:40.8983478Z return self._bootstrap(parent_sentinel) 2023-01-11T22:17:40.8983880Z File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap 2023-01-11T22:17:40.8984222Z self.run() 2023-01-11T22:17:40.8984560Z File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run 2023-01-11T22:17:40.8984949Z self._target(*self._args, **self._kwargs) 2023-01-11T22:17:40.8985459Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 824, in _run 2023-01-11T22:17:40.8985877Z self.run_test(test_name, pipe) 2023-01-11T22:17:40.8986406Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-01-11T22:17:40.8986808Z getattr(self, test_name)() 2023-01-11T22:17:40.8987325Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 536, in wrapper 2023-01-11T22:17:40.8987723Z fn() 2023-01-11T22:17:40.8988210Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 167, in wrapper 2023-01-11T22:17:40.8988613Z return func(*args, **kwargs) 2023-01-11T22:17:40.8989036Z File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_apply.py", line 98, in test_apply_in_summon_raises_error 2023-01-11T22:17:40.8989503Z transformer.apply(self._init_linear_weights) 2023-01-11T22:17:40.8990061Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 491, in apply 2023-01-11T22:17:40.8990524Z self._assert_state(TrainingState.IDLE) 2023-01-11T22:17:40.8991102Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 930, in _assert_state 2023-01-11T22:17:40.8991540Z traceback.print_stack() 2023-01-11T22:17:40.8991799Z dist init r=1, world=2 2023-01-11T22:17:40.8992059Z dist init r=0, world=2 2023-01-11T22:17:40.8992388Z Asserting FSDP instance is: FullyShardedDataParallel( 2023-01-11T22:17:40.8992777Z (_fsdp_wrapped_module): TransformerWithSharedParams( 2023-01-11T22:17:40.8993144Z (embed_tokens): Embedding(23, 16) 2023-01-11T22:17:40.8993437Z (transformer): Transformer( 2023-01-11T22:17:40.8993722Z (encoder): TransformerEncoder( 2023-01-11T22:17:40.8994020Z (layers): ModuleList( 2023-01-11T22:17:40.8994395Z (0-1): 2 x FullyShardedDataParallel( 2023-01-11T22:17:40.8994819Z (_fsdp_wrapped_module): TransformerEncoderLayer( 2023-01-11T22:17:40.8995185Z (self_attn): MultiheadAttention( 2023-01-11T22:17:40.8995607Z (out_proj): NonDynamicallyQuantizableLinear(in_features=16, out_features=16, bias=True) 2023-01-11T22:17:40.8995972Z ) 2023-01-11T22:17:40.8996279Z (linear1): Linear(in_features=16, out_features=8, bias=True) 2023-01-11T22:17:40.8996644Z (dropout): Dropout(p=0.1, inplace=False) 2023-01-11T22:17:40.8997011Z (linear2): Linear(in_features=8, out_features=16, bias=True) 2023-01-11T22:17:40.8997469Z (norm1): LayerNorm((16,), eps=1e-05, elementwise_affine=True) 2023-01-11T22:17:40.8997931Z (norm2): LayerNorm((16,), eps=1e-05, elementwise_affine=True) 2023-01-11T22:17:40.8998295Z (dropout1): Dropout(p=0.1, inplace=False) 2023-01-11T22:17:40.8998618Z (dropout2): Dropout(p=0.1, inplace=False) 2023-01-11T22:17:40.8998900Z ) 2023-01-11T22:17:40.8999130Z ) 2023-01-11T22:17:40.8999339Z ) 2023-01-11T22:17:40.8999712Z (norm): LayerNorm((16,), eps=1e-05, elementwise_affine=True) 2023-01-11T22:17:40.9000022Z ) 2023-01-11T22:17:40.9000266Z (decoder): TransformerDecoder( 2023-01-11T22:17:40.9000563Z (layers): ModuleList( 2023-01-11T22:17:40.9000932Z (0-1): 2 x FullyShardedDataParallel( 2023-01-11T22:17:40.9001289Z (_fsdp_wrapped_module): TransformerDecoderLayer( 2023-01-11T22:17:40.9001712Z (self_attn): MultiheadAttention( 2023-01-11T22:17:40.9002144Z (out_proj): NonDynamicallyQuantizableLinear(in_features=16, out_features=16, bias=True) 2023-01-11T22:17:40.9002525Z ) 2023-01-11T22:17:40.9002789Z (multihead_attn): MultiheadAttention( 2023-01-11T22:17:40.9003223Z (out_proj): NonDynamicallyQuantizableLinear(in_features=16, out_features=16, bias=True) 2023-01-11T22:17:40.9003609Z ) 2023-01-11T22:17:40.9003904Z (linear1): Linear(in_features=16, out_features=8, bias=True) 2023-01-11T22:17:40.9004268Z (dropout): Dropout(p=0.1, inplace=False) 2023-01-11T22:17:40.9004632Z (linear2): Linear(in_features=8, out_features=16, bias=True) 2023-01-11T22:17:40.9005086Z (norm1): LayerNorm((16,), eps=1e-05, elementwise_affine=True) 2023-01-11T22:17:40.9005557Z (norm2): LayerNorm((16,), eps=1e-05, elementwise_affine=True) 2023-01-11T22:17:40.9006020Z (norm3): LayerNorm((16,), eps=1e-05, elementwise_affine=True) 2023-01-11T22:17:40.9006379Z (dropout1): Dropout(p=0.1, inplace=False) 2023-01-11T22:17:40.9006700Z (dropout2): Dropout(p=0.1, inplace=False) 2023-01-11T22:17:40.9007036Z (dropout3): Dropout(p=0.1, inplace=False) 2023-01-11T22:17:40.9007315Z ) 2023-01-11T22:17:40.9007525Z ) 2023-01-11T22:17:40.9007749Z ) 2023-01-11T22:17:40.9008125Z (norm): LayerNorm((16,), eps=1e-05, elementwise_affine=True) 2023-01-11T22:17:40.9008420Z ) 2023-01-11T22:17:40.9008636Z ) 2023-01-11T22:17:40.9008936Z (output_proj): Linear(in_features=16, out_features=23, bias=True) 2023-01-11T22:17:40.9009425Z (bn): BatchNorm1d(2, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) 2023-01-11T22:17:40.9009756Z ) 2023-01-11T22:17:40.9009966Z ) 2023-01-11T22:17:40.9010523Z ERROR: expected to be in states [] but current state is TrainingState.SUMMON_FULL_PARAMS 2023-01-11T22:17:40.9010941Z ok (5.666s) 2023-01-11T22:17:40.9011226Z test_nested_module_apply (__main__.TestApply) 2023-01-11T22:17:40.9011858Z Tests that ``apply()`` modifies parameter values in-place on a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57937 2023-01-11T22:17:40.9012405Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57938 2023-01-11T22:17:40.9013034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:40.9013608Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:40.9014179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:40.9014676Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:40.9015271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:40.9015740Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:40.9016299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:40.9016784Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:40.9017254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:17:40.9017774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:17:40.9018437Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:17:40.9019158Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:17:40.9019803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:17:40.9020286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:17:40.9021573Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:17:40.9022451Z warnings.warn( 2023-01-11T22:17:40.9023606Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:17:40.9024457Z warnings.warn( 2023-01-11T22:17:40.9024697Z dist init r=1, world=2 2023-01-11T22:17:40.9024951Z dist init r=0, world=2 2023-01-11T22:17:40.9025196Z ok (4.110s) 2023-01-11T22:17:40.9025492Z test_transformer_module_apply (__main__.TestApply) 2023-01-11T22:17:40.9026115Z Tests that ``apply()`` modifies parameter values in-place on an ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58016 2023-01-11T22:17:40.9026690Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58017 2023-01-11T22:17:40.9027318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:40.9027770Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:40.9028362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:40.9028854Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:40.9029447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:40.9029892Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:40.9030473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:40.9031033Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:40.9031479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:17:40.9031996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:17:40.9032678Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:17:40.9033402Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:17:40.9033935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:17:40.9034429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:17:40.9035704Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:17:40.9036577Z warnings.warn( 2023-01-11T22:17:40.9037783Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:17:40.9038635Z warnings.warn( 2023-01-11T22:17:40.9038893Z dist init r=1, world=2 2023-01-11T22:17:40.9039149Z dist init r=0, world=2 2023-01-11T22:17:40.9039394Z ok (4.311s) 2023-01-11T22:17:40.9039531Z 2023-01-11T22:17:40.9039801Z ---------------------------------------------------------------------- 2023-01-11T22:17:40.9040138Z Ran 3 tests in 14.087s 2023-01-11T22:17:40.9040306Z 2023-01-11T22:17:40.9040400Z OK 2023-01-11T22:17:40.9040519Z 2023-01-11T22:17:40.9040642Z Generating XML reports... 2023-01-11T22:17:40.9041216Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_apply/TEST-TestApply-20230111221726.xml 2023-01-11T22:17:40.9041587Z 2023-01-11T22:17:40.9041917Z ##[endgroup] 2023-01-11T22:17:40.9042487Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_apply (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_apply_0b_ajz4z) 2023-01-11T22:17:40.9042876Z 2023-01-11T22:17:40.9043156Z Running distributed/_tensor/test_tp_sharding_ops ... [2023-01-11 22:17:40.896405] 2023-01-11T22:17:40.9043883Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_tp_sharding_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:17:40.896697] 2023-01-11T22:17:59.1959927Z 2023-01-11T22:17:59.1960461Z Expand the folded group to see the log file of distributed/_tensor/test_tp_sharding_ops 2023-01-11T22:17:59.1961709Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_tp_sharding_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_tp_sharding_ops_su3yyaij) 2023-01-11T22:17:59.1962083Z 2023-01-11T22:17:59.1962199Z Running tests... 2023-01-11T22:17:59.1962720Z ---------------------------------------------------------------------- 2023-01-11T22:17:59.1963284Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_tp_sharding_ops 2023-01-11T22:17:59.1963819Z test_replicated_permute (__main__.TPShardingOpsTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:17:59.1964319Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58130 2023-01-11T22:17:59.1965075Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58131 2023-01-11T22:17:59.1965519Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 58132 2023-01-11T22:17:59.1965973Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 58133 2023-01-11T22:17:59.1966631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:59.1967100Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:59.1967698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:59.1968187Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:59.1968775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:59.1969231Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:59.1969810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:59.1972392Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:59.1973093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:59.1973790Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:59.1974406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:59.1974882Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:59.1975448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:59.1975890Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:59.1976473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:59.1976930Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:59.1977367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:17:59.1977841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T22:17:59.1978316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T22:17:59.1978759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:17:59.1979134Z skip: Need at least 4 CUDA devices (4.026s) 2023-01-11T22:17:59.1979599Z test_sharded_cat (__main__.TPShardingOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58266 2023-01-11T22:17:59.1980113Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58267 2023-01-11T22:17:59.1980547Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 58268 2023-01-11T22:17:59.1980980Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 58269 2023-01-11T22:17:59.1981604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:59.1982037Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:59.1982613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:59.1983078Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:59.1983645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:59.1984069Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:59.1984756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:59.1985217Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:59.1985771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:59.1986212Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:59.1986780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:59.1987238Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:59.1987791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:59.1988239Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:59.1988813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:59.1989275Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:59.1989688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:17:59.1990161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T22:17:59.1990693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:17:59.1991148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T22:17:59.1991535Z skip: Need at least 4 CUDA devices (2.410s) 2023-01-11T22:17:59.1992000Z test_sharded_permute (__main__.TPShardingOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58402 2023-01-11T22:17:59.1992513Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58403 2023-01-11T22:17:59.1992943Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 58404 2023-01-11T22:17:59.1993374Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 58405 2023-01-11T22:17:59.1994041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:59.1994492Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:59.1995037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:59.1995482Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:59.1996055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:59.1996520Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:59.1997084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:59.1997551Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:59.1998125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:59.1998554Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:59.1999123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:59.1999589Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:59.2000149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:59.2000570Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:59.2001134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:59.2001669Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:59.2002101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T22:17:59.2002550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:17:59.2003008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:17:59.2003473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T22:17:59.2003843Z skip: Need at least 4 CUDA devices (2.409s) 2023-01-11T22:17:59.2004304Z test_sharded_split (__main__.TPShardingOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58538 2023-01-11T22:17:59.2004818Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58539 2023-01-11T22:17:59.2005263Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 58540 2023-01-11T22:17:59.2005687Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 58541 2023-01-11T22:17:59.2006287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:59.2006734Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:59.2007365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:59.2007838Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:59.2008415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:59.2008856Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:59.2009405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:59.2009874Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:59.2010708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:59.2011157Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:59.2011713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:59.2012184Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:59.2012754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:59.2013175Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:59.2013741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:59.2014204Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:59.2014635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:17:59.2015084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T22:17:59.2015550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:17:59.2016009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T22:17:59.2016381Z skip: Need at least 4 CUDA devices (2.410s) 2023-01-11T22:17:59.2016850Z test_sharded_transpose (__main__.TPShardingOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58674 2023-01-11T22:17:59.2017372Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58675 2023-01-11T22:17:59.2017815Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 58676 2023-01-11T22:17:59.2018421Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 58677 2023-01-11T22:17:59.2019022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:59.2019470Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:59.2020018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:59.2020490Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:59.2021061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:59.2021506Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:59.2022055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:59.2022526Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:59.2023096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:59.2023535Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:59.2024080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:59.2024619Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:59.2025202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:59.2025625Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:59.2026191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:59.2026649Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:59.2027078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T22:17:59.2027527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T22:17:59.2027986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:17:59.2028444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:17:59.2028820Z skip: Need at least 4 CUDA devices (2.410s) 2023-01-11T22:17:59.2029281Z test_sharded_view (__main__.TPShardingOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58810 2023-01-11T22:17:59.2029797Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58811 2023-01-11T22:17:59.2030240Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 58812 2023-01-11T22:17:59.2030661Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 58813 2023-01-11T22:17:59.2031267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:59.2031712Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:59.2032278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:59.2032729Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:59.2033310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:59.2033754Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:59.2034302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:59.2034761Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:59.2035408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:59.2035847Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:59.2036392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:59.2036853Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:59.2037424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:17:59.2037846Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:17:59.2038410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:17:59.2038869Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:17:59.2039298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T22:17:59.2039756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T22:17:59.2040209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:17:59.2040668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:17:59.2041039Z skip: Need at least 4 CUDA devices (2.309s) 2023-01-11T22:17:59.2041235Z 2023-01-11T22:17:59.2041569Z ---------------------------------------------------------------------- 2023-01-11T22:17:59.2041908Z Ran 6 tests in 15.976s 2023-01-11T22:17:59.2042072Z 2023-01-11T22:17:59.2042178Z OK (skipped=6) 2023-01-11T22:17:59.2042332Z 2023-01-11T22:17:59.2042439Z Generating XML reports... 2023-01-11T22:17:59.2043047Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_tp_sharding_ops/TEST-TPShardingOpsTest-20230111221742.xml 2023-01-11T22:17:59.2043414Z 2023-01-11T22:17:59.2043867Z ##[endgroup] 2023-01-11T22:17:59.2044468Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_tp_sharding_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_tp_sharding_ops_su3yyaij) 2023-01-11T22:17:59.2044837Z 2023-01-11T22:17:59.2045117Z Running distributed/fsdp/test_fsdp_hybrid_shard ... [2023-01-11 22:17:59.196186] 2023-01-11T22:17:59.2045814Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_hybrid_shard.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:17:59.196475] 2023-01-11T22:18:20.7756773Z 2023-01-11T22:18:20.7757516Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_hybrid_shard 2023-01-11T22:18:20.7758456Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_hybrid_shard (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_hybrid_shard_2l0ikv47) 2023-01-11T22:18:20.7758823Z 2023-01-11T22:18:20.7758918Z Running tests... 2023-01-11T22:18:20.7759736Z ---------------------------------------------------------------------- 2023-01-11T22:18:20.7760315Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_hybrid_shard 2023-01-11T22:18:20.7763487Z test_fsdp_hybrid_shard_basic_setup (__main__.TestFSDPHybridShard) 2023-01-11T22:18:20.7763998Z Tests basic functionality of HYBRID_SHARD and _HYBRID_SHARD_ZERO2: ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:18:20.7764513Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58981 2023-01-11T22:18:20.7764961Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58982 2023-01-11T22:18:20.7765632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:18:20.7766077Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:18:20.7766659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:18:20.7767413Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:18:20.7767994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:18:20.7768444Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:18:20.7769022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:18:20.7769490Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:18:20.7769923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:18:20.7770847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:18:20.7771879Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:18:20.7775144Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:18:20.7776216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:18:20.7777061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:18:20.7777947Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T22:18:20.7779026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T22:18:20.7780378Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:18:20.7780903Z INFO:torch.distributed.distributed_c10d:Rank 1 is assigned to subgroup [0, 1] 2023-01-11T22:18:20.7781376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2023-01-11T22:18:20.7782038Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:18:20.7782527Z INFO:torch.distributed.distributed_c10d:Rank 0 is assigned to subgroup [0, 1] 2023-01-11T22:18:20.7782995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2023-01-11T22:18:20.7783641Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2023-01-11T22:18:20.7784321Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2023-01-11T22:18:20.7784829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2023-01-11T22:18:20.7785310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2023-01-11T22:18:20.7785955Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2023-01-11T22:18:20.7786619Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2023-01-11T22:18:20.7787140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1 2023-01-11T22:18:20.7787629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0 2023-01-11T22:18:20.7788272Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 2 nodes. 2023-01-11T22:18:20.7788762Z INFO:torch.distributed.distributed_c10d:Rank 0 is assigned to subgroup [0, 1] 2023-01-11T22:18:20.7789227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 0 2023-01-11T22:18:20.7789864Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 2 nodes. 2023-01-11T22:18:20.7790492Z INFO:torch.distributed.distributed_c10d:Rank 1 is assigned to subgroup [0, 1] 2023-01-11T22:18:20.7790942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 1 2023-01-11T22:18:20.7791590Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:6 with 2 nodes. 2023-01-11T22:18:20.7792120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 1 2023-01-11T22:18:20.7792757Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:6 with 2 nodes. 2023-01-11T22:18:20.7793264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 0 2023-01-11T22:18:20.7793897Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:7 with 2 nodes. 2023-01-11T22:18:20.7794566Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:7 with 2 nodes. 2023-01-11T22:18:20.7795078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 0 2023-01-11T22:18:20.7795558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 1 2023-01-11T22:18:20.7796198Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:8 with 2 nodes. 2023-01-11T22:18:20.7796763Z INFO:torch.distributed.distributed_c10d:Rank 1 is assigned to subgroup [0, 1] 2023-01-11T22:18:20.7797228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 1 2023-01-11T22:18:20.7797867Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:8 with 2 nodes. 2023-01-11T22:18:20.7798372Z INFO:torch.distributed.distributed_c10d:Rank 0 is assigned to subgroup [0, 1] 2023-01-11T22:18:20.7798840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 0 2023-01-11T22:18:20.7799467Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:9 with 2 nodes. 2023-01-11T22:18:20.7799998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 0 2023-01-11T22:18:20.7800638Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:9 with 2 nodes. 2023-01-11T22:18:20.7801155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 1 2023-01-11T22:18:20.7801778Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:10 with 2 nodes. 2023-01-11T22:18:20.7802447Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:10 with 2 nodes. 2023-01-11T22:18:20.7802976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 1 2023-01-11T22:18:20.7803446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 0 2023-01-11T22:18:20.7804090Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:11 with 2 nodes. 2023-01-11T22:18:20.7804590Z INFO:torch.distributed.distributed_c10d:Rank 0 is assigned to subgroup [0, 1] 2023-01-11T22:18:20.7805065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 0 2023-01-11T22:18:20.7805688Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:11 with 2 nodes. 2023-01-11T22:18:20.7806191Z INFO:torch.distributed.distributed_c10d:Rank 1 is assigned to subgroup [0, 1] 2023-01-11T22:18:20.7806659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 1 2023-01-11T22:18:20.7807474Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:12 with 2 nodes. 2023-01-11T22:18:20.7808000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 1 2023-01-11T22:18:20.7808640Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:12 with 2 nodes. 2023-01-11T22:18:20.7809154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 0 2023-01-11T22:18:20.7809800Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:13 with 2 nodes. 2023-01-11T22:18:20.7810850Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:13 with 2 nodes. 2023-01-11T22:18:20.7811910Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7813237Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7814567Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7815810Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7817034Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7818264Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7819486Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7820717Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7821937Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7823159Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7824463Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7825684Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7826885Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7828103Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7829377Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7830606Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7831817Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7833039Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7834262Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7835476Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7836698Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7837891Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7839171Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7840388Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7841124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 0 2023-01-11T22:18:20.7841620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 1 2023-01-11T22:18:20.7842259Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:14 with 2 nodes. 2023-01-11T22:18:20.7842764Z INFO:torch.distributed.distributed_c10d:Rank 1 is assigned to subgroup [0, 1] 2023-01-11T22:18:20.7843237Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 1 2023-01-11T22:18:20.7843955Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:14 with 2 nodes. 2023-01-11T22:18:20.7844448Z INFO:torch.distributed.distributed_c10d:Rank 0 is assigned to subgroup [0, 1] 2023-01-11T22:18:20.7844910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 0 2023-01-11T22:18:20.7845550Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:15 with 2 nodes. 2023-01-11T22:18:20.7846076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 0 2023-01-11T22:18:20.7846704Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:15 with 2 nodes. 2023-01-11T22:18:20.7847224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 1 2023-01-11T22:18:20.7847871Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:16 with 2 nodes. 2023-01-11T22:18:20.7848538Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:16 with 2 nodes. 2023-01-11T22:18:20.7849051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 1 2023-01-11T22:18:20.7849536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 0 2023-01-11T22:18:20.7850180Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:17 with 2 nodes. 2023-01-11T22:18:20.7850965Z INFO:torch.distributed.distributed_c10d:Rank 0 is assigned to subgroup [0, 1] 2023-01-11T22:18:20.7851437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 0 2023-01-11T22:18:20.7852090Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:17 with 2 nodes. 2023-01-11T22:18:20.7852593Z INFO:torch.distributed.distributed_c10d:Rank 1 is assigned to subgroup [0, 1] 2023-01-11T22:18:20.7853045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 1 2023-01-11T22:18:20.7853686Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:18 with 2 nodes. 2023-01-11T22:18:20.7854244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 1 2023-01-11T22:18:20.7854993Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:18 with 2 nodes. 2023-01-11T22:18:20.7855499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 0 2023-01-11T22:18:20.7856144Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:19 with 2 nodes. 2023-01-11T22:18:20.7856823Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:19 with 2 nodes. 2023-01-11T22:18:20.7857347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:20 to store for rank: 0 2023-01-11T22:18:20.7857814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:20 to store for rank: 1 2023-01-11T22:18:20.7858458Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:20 with 2 nodes. 2023-01-11T22:18:20.7858958Z INFO:torch.distributed.distributed_c10d:Rank 1 is assigned to subgroup [0, 1] 2023-01-11T22:18:20.7859412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:21 to store for rank: 1 2023-01-11T22:18:20.7860051Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:20 with 2 nodes. 2023-01-11T22:18:20.7860549Z INFO:torch.distributed.distributed_c10d:Rank 0 is assigned to subgroup [0, 1] 2023-01-11T22:18:20.7861084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:21 to store for rank: 0 2023-01-11T22:18:20.7861717Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:21 with 2 nodes. 2023-01-11T22:18:20.7862233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:22 to store for rank: 0 2023-01-11T22:18:20.7862864Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:21 with 2 nodes. 2023-01-11T22:18:20.7863388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:22 to store for rank: 1 2023-01-11T22:18:20.7864014Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:22 with 2 nodes. 2023-01-11T22:18:20.7864680Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:22 with 2 nodes. 2023-01-11T22:18:20.7865205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:23 to store for rank: 1 2023-01-11T22:18:20.7865691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:23 to store for rank: 0 2023-01-11T22:18:20.7866312Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:23 with 2 nodes. 2023-01-11T22:18:20.7866809Z INFO:torch.distributed.distributed_c10d:Rank 0 is assigned to subgroup [0, 1] 2023-01-11T22:18:20.7867275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:24 to store for rank: 0 2023-01-11T22:18:20.7867897Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:23 with 2 nodes. 2023-01-11T22:18:20.7868394Z INFO:torch.distributed.distributed_c10d:Rank 1 is assigned to subgroup [0, 1] 2023-01-11T22:18:20.7868856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:24 to store for rank: 1 2023-01-11T22:18:20.7869492Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:24 with 2 nodes. 2023-01-11T22:18:20.7869996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:25 to store for rank: 1 2023-01-11T22:18:20.7870633Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:24 with 2 nodes. 2023-01-11T22:18:20.7871159Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:25 to store for rank: 0 2023-01-11T22:18:20.7871883Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:25 with 2 nodes. 2023-01-11T22:18:20.7872537Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:25 with 2 nodes. 2023-01-11T22:18:20.7873558Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7874792Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7876029Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7877328Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7878560Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7879788Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7880989Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7882207Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7883427Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7884652Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7885864Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7887076Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7888367Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7889577Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7891083Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7892374Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7893612Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7894824Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7896032Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7897242Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7898448Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7899676Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7900881Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7902082Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7902766Z dist init r=1, world=2 2023-01-11T22:18:20.7903036Z 1 created process group for [1] 2023-01-11T22:18:20.7903323Z 1 created process group for [1] 2023-01-11T22:18:20.7903583Z 1 created process group for [1] 2023-01-11T22:18:20.7903865Z 1 created process group for [1] 2023-01-11T22:18:20.7904145Z 1 created process group for [1] 2023-01-11T22:18:20.7904407Z 1 created process group for [1] 2023-01-11T22:18:20.7904684Z 1 created process group for [1] 2023-01-11T22:18:20.7904965Z 1 created process group for [1] 2023-01-11T22:18:20.7905207Z dist init r=0, world=2 2023-01-11T22:18:20.7905467Z 0 created process group for [0] 2023-01-11T22:18:20.7905742Z 0 created process group for [0] 2023-01-11T22:18:20.7906005Z 0 created process group for [0] 2023-01-11T22:18:20.7906277Z 0 created process group for [0] 2023-01-11T22:18:20.7906553Z 0 created process group for [0] 2023-01-11T22:18:20.7906811Z 0 created process group for [0] 2023-01-11T22:18:20.7907085Z 0 created process group for [0] 2023-01-11T22:18:20.7907360Z 0 created process group for [0] 2023-01-11T22:18:20.7907592Z ok (7.385s) 2023-01-11T22:18:20.7908098Z test_hybrid_shard_pg_mismatch_raises (__main__.TestFSDPHybridShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59156 2023-01-11T22:18:20.7908651Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59157 2023-01-11T22:18:20.7909267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:18:20.7909698Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:18:20.7910268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:18:20.7910740Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:18:20.7911312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:18:20.7911737Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:18:20.7912304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:18:20.7912768Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:18:20.7913200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:18:20.7913688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:18:20.7914341Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:18:20.7915032Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:18:20.7915533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:18:20.7916002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:18:20.7916484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T22:18:20.7916970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T22:18:20.7917607Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:18:20.7918287Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:18:20.7918811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2023-01-11T22:18:20.7919345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2023-01-11T22:18:20.7919991Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2023-01-11T22:18:20.7920669Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2023-01-11T22:18:20.7921198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2023-01-11T22:18:20.7921667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2023-01-11T22:18:20.7922308Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2023-01-11T22:18:20.7922980Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2023-01-11T22:18:20.7923511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0 2023-01-11T22:18:20.7923979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1 2023-01-11T22:18:20.7924616Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 2 nodes. 2023-01-11T22:18:20.7925702Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7926607Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 2 nodes. 2023-01-11T22:18:20.7927627Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:18:20.7928230Z dist init r=1, world=2 2023-01-11T22:18:20.7928479Z dist init r=0, world=2 2023-01-11T22:18:20.7928715Z ok (4.012s) 2023-01-11T22:18:20.7929146Z test_invalid_pg_specification_raises (__main__.TestFSDPHybridShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59243 2023-01-11T22:18:20.7929690Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59244 2023-01-11T22:18:20.7930573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:18:20.7931041Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:18:20.7931604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:18:20.7932076Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:18:20.7932649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:18:20.7933072Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:18:20.7933640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:18:20.7934105Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:18:20.7934550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:18:20.7935023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:18:20.7935670Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:18:20.7936457Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:18:20.7936971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:18:20.7937419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:18:20.7937766Z dist init r=1, world=2 2023-01-11T22:18:20.7938019Z dist init r=0, world=2 2023-01-11T22:18:20.7938240Z ok (3.910s) 2023-01-11T22:18:20.7938698Z test_raises_manual_wrap_hybrid_shard_when_none_policy (__main__.TestFSDPHybridShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59322 2023-01-11T22:18:20.7939252Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59323 2023-01-11T22:18:20.7939857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:18:20.7940291Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:18:20.7940859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:18:20.7941324Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:18:20.7941879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:18:20.7942398Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:18:20.7942975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:18:20.7943436Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:18:20.7943866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:18:20.7944358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:18:20.7945013Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:18:20.7945698Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:18:20.7946198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:18:20.7946665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:18:20.7947023Z dist init r=1, world=2 2023-01-11T22:18:20.7947253Z dist init r=0, world=2 2023-01-11T22:18:20.7947493Z ok (3.910s) 2023-01-11T22:18:20.7947642Z 2023-01-11T22:18:20.7947911Z ---------------------------------------------------------------------- 2023-01-11T22:18:20.7948221Z Ran 4 tests in 19.218s 2023-01-11T22:18:20.7948380Z 2023-01-11T22:18:20.7948471Z OK 2023-01-11T22:18:20.7948606Z 2023-01-11T22:18:20.7948730Z Generating XML reports... 2023-01-11T22:18:20.7949338Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_hybrid_shard/TEST-TestFSDPHybridShard-20230111221801.xml 2023-01-11T22:18:20.7949686Z 2023-01-11T22:18:20.7950083Z ##[endgroup] 2023-01-11T22:18:20.7950698Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_hybrid_shard (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_hybrid_shard_2l0ikv47) 2023-01-11T22:18:20.7951069Z 2023-01-11T22:18:20.7951340Z Running distributed/test_dynamo_distributed ... [2023-01-11 22:18:20.776191] 2023-01-11T22:18:20.7952002Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_dynamo_distributed.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:18:20.776477] 2023-01-11T22:18:44.7285300Z 2023-01-11T22:18:44.7286066Z Expand the folded group to see the log file of distributed/test_dynamo_distributed 2023-01-11T22:18:44.7287244Z ##[group]PRINTING LOG FILE of distributed/test_dynamo_distributed (/var/lib/jenkins/workspace/test/test-reports/distributed-test_dynamo_distributed_wj6tgrhi) 2023-01-11T22:18:44.7289315Z 2023-01-11T22:18:44.7290043Z Running tests... 2023-01-11T22:18:44.7291110Z ---------------------------------------------------------------------- 2023-01-11T22:18:44.7291589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:18:44.7293467Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2023-01-11T22:18:44.7294707Z Test results will be stored in test-reports/python-unittest/distributed.test_dynamo_distributed 2023-01-11T22:18:44.7295432Z test_aot_autograd (__main__.TestDistributed) 2023-01-11T22:18:44.7296288Z Explicitly check AotAutograd family of compilers work, ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:18:44.7297184Z [2023-01-11 22:18:25,604] torch._dynamo.variables.torch: [WARNING] Profiler will be ignored 2023-01-11T22:18:44.7297512Z ok (2.965s) 2023-01-11T22:18:44.7297832Z test_custom_layer (__main__.TestDistributed) 2023-01-11T22:18:44.7298977Z Just ensures that the appropriate number of splits happen (based on ... [2023-01-11 22:18:26,087] torch._dynamo.variables.torch: [WARNING] Profiler will be ignored 2023-01-11T22:18:44.7299950Z frames [('total', 4), ('ok', 4)] 2023-01-11T22:18:44.7300791Z inline_call [('call_function UserDefinedObjectVariable(instancemethod) [] {}', 1)] 2023-01-11T22:18:44.7301634Z unimplemented [] 2023-01-11T22:18:44.7302106Z graph_break [('call_function UserDefinedObjectVariable(instancemethod) [] {}', 1)] 2023-01-11T22:18:44.7302857Z stats [('calls_captured', 8), ('fusions_possible', 7), ('unique_graphs', 1)] 2023-01-11T22:18:44.7303595Z aot_autograd [('total', 3), ('ok', 3)] 2023-01-11T22:18:44.7304138Z frames [('total', 2), ('ok', 2)] 2023-01-11T22:18:44.7305032Z inline_call [('call_function UserDefinedObjectVariable(instancemethod) [] {}', 1)] 2023-01-11T22:18:44.7305605Z unimplemented [] 2023-01-11T22:18:44.7306300Z graph_break [('call_function UserDefinedObjectVariable(instancemethod) [] {}', 1)] 2023-01-11T22:18:44.7307302Z stats [('calls_captured', 7), ('fusions_possible', 6), ('unique_graphs', 1)] 2023-01-11T22:18:44.7307896Z ok (0.084s) 2023-01-11T22:18:44.7308708Z test_ddp_baseline_aot_eager (__main__.TestDistributed) ... frames [('total', 1), ('ok', 1)] 2023-01-11T22:18:44.7309639Z stats [('calls_captured', 8), ('fusions_possible', 7), ('unique_graphs', 1)] 2023-01-11T22:18:44.7310341Z aot_autograd [('total', 1), ('ok', 1)] 2023-01-11T22:18:44.7310751Z ok (0.736s) 2023-01-11T22:18:44.7311349Z test_ddp_baseline_inductor (__main__.TestDistributed) ... skip: Inductor+gpu needs triton and recent GPU arch (0.001s) 2023-01-11T22:18:44.7311883Z test_empty_graph_inductor (__main__.TestDistributed) ... skip: Inductor+gpu needs triton and recent GPU arch (0.000s) 2023-01-11T22:18:44.7312431Z test_fsdp_orig_params_assert (__main__.TestDistributed) ... frames [('total', 1)] 2023-01-11T22:18:44.7312772Z ok (1.474s) 2023-01-11T22:18:44.7313053Z test_graph_split (__main__.TestDistributed) 2023-01-11T22:18:44.7313694Z Just ensures that the appropriate number of splits happen (based on ... [2023-01-11 22:18:28,869] torch._dynamo.variables.torch: [WARNING] Profiler will be ignored 2023-01-11T22:18:44.7314163Z frames [('total', 2), ('ok', 2)] 2023-01-11T22:18:44.7314618Z inline_call [('call_function UserDefinedObjectVariable(instancemethod) [] {}', 1)] 2023-01-11T22:18:44.7314977Z unimplemented [] 2023-01-11T22:18:44.7315401Z graph_break [('call_function UserDefinedObjectVariable(instancemethod) [] {}', 1)] 2023-01-11T22:18:44.7315916Z stats [('calls_captured', 8), ('fusions_possible', 7), ('unique_graphs', 1)] 2023-01-11T22:18:44.7316227Z ok (0.600s) 2023-01-11T22:18:44.7316523Z test_graph_split_inductor (__main__.TestDistributed) 2023-01-11T22:18:44.7316938Z Same as above, but using inductor backend. ... skip: Inductor+gpu needs triton and recent GPU arch (0.001s) 2023-01-11T22:18:44.7317524Z test_ignored_parameters (__main__.TestDistributed) 2023-01-11T22:18:44.7318206Z Verifies ddp graph-split logic ignores parameters marked to ignore on DDP module. ... [2023-01-11 22:18:28,984] torch._dynamo.variables.torch: [WARNING] Profiler will be ignored 2023-01-11T22:18:44.7318698Z frames [('total', 2), ('ok', 2)] 2023-01-11T22:18:44.7319154Z inline_call [('call_function UserDefinedObjectVariable(instancemethod) [] {}', 1)] 2023-01-11T22:18:44.7319508Z unimplemented [] 2023-01-11T22:18:44.7319952Z graph_break [('call_function UserDefinedObjectVariable(instancemethod) [] {}', 1)] 2023-01-11T22:18:44.7320447Z stats [('calls_captured', 7), ('fusions_possible', 6), ('unique_graphs', 1)] 2023-01-11T22:18:44.7320765Z ok (0.080s) 2023-01-11T22:18:44.7321043Z test_no_split (__main__.TestDistributed) 2023-01-11T22:18:44.7321647Z Ensures the DDPOptimizer returns a correct, compiled module without ... [2023-01-11 22:18:29,054] torch._dynamo.variables.torch: [WARNING] Profiler will be ignored 2023-01-11T22:18:44.7322141Z frames [('total', 2), ('ok', 2)] 2023-01-11T22:18:44.7322588Z inline_call [('call_function UserDefinedObjectVariable(instancemethod) [] {}', 1)] 2023-01-11T22:18:44.7322928Z unimplemented [] 2023-01-11T22:18:44.7323371Z graph_break [('call_function UserDefinedObjectVariable(instancemethod) [] {}', 1)] 2023-01-11T22:18:44.7323882Z stats [('calls_captured', 8), ('fusions_possible', 7), ('unique_graphs', 1)] 2023-01-11T22:18:44.7324195Z ok (0.056s) 2023-01-11T22:18:44.7324778Z test_ddp_baseline_aot_eager_multiprocess (__main__.TestDistributedMultiProc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59517 2023-01-11T22:18:44.7325336Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59518 2023-01-11T22:18:44.7325947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:18:44.7326406Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:18:44.7326970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:18:44.7327451Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:18:44.7328026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:18:44.7328476Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:18:44.7329032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:18:44.7329500Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:18:44.7329938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:18:44.7330775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:18:44.7331264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:18:44.7331758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:18:44.7332424Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:18:44.7333103Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:18:44.7333500Z ok (5.817s) 2023-01-11T22:18:44.7333937Z test_fsdp_aot_eager (__main__.TestDistributedMultiProc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59607 2023-01-11T22:18:44.7334479Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59608 2023-01-11T22:18:44.7335067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:18:44.7335622Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:18:44.7336197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:18:44.7336650Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:18:44.7337229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:18:44.7337674Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:18:44.7338245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:18:44.7338690Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:18:44.7339128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:18:44.7339604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:18:44.7340092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:18:44.7340570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:18:44.7341222Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:18:44.7341989Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:18:44.7342395Z ok (6.316s) 2023-01-11T22:18:44.7342774Z test_fsdp_inductor (__main__.TestDistributedMultiProc) ... skip: Inductor+gpu needs triton and recent GPU arch (0.001s) 2023-01-11T22:18:44.7343370Z test_hf_bert_ddp_aot_eager (__main__.TestDistributedMultiProc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59697 2023-01-11T22:18:44.7343909Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59698 2023-01-11T22:18:44.7344513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:18:44.7344964Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:18:44.7345534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:18:44.7346007Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:18:44.7346564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:18:44.7347010Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:18:44.7347592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:18:44.7348043Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:18:44.7348461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:18:44.7348940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:18:44.7349348Z skip: Test skipped due to missing import (2.809s) 2023-01-11T22:18:44.7349794Z test_hf_bert_ddp_inductor (__main__.TestDistributedMultiProc) ... skip: Inductor+gpu needs triton and recent GPU arch (0.001s) 2023-01-11T22:18:44.7350354Z test_hf_bert_fsdp (__main__.TestDistributedMultiProc) ... skip: Inductor+gpu needs triton and recent GPU arch (0.001s) 2023-01-11T22:18:44.7350907Z test_hf_bert_ddp_aot_eager (__main__.TestFakeDistributedSingleProc) ... skip: Unable to import transformers (0.001s) 2023-01-11T22:18:44.7351486Z test_hf_bert_ddp_inductor (__main__.TestFakeDistributedSingleProc) ... skip: Inductor+gpu needs triton and recent GPU arch (0.000s) 2023-01-11T22:18:44.7352116Z test_issue90375 (__main__.TestFakeDistributedSingleProc) ... frames [('total', 4), ('ok', 3)] 2023-01-11T22:18:44.7352732Z inline_call [('inline in skipfiles: helper /opt/conda/lib/python3.10/contextlib.py', 1)] 2023-01-11T22:18:44.7353339Z unimplemented [('SETUP_WITH UserDefinedObjectVariable(_GeneratorContextManager)', 1), ('generator', 1)] 2023-01-11T22:18:44.7353922Z graph_break [('inline in skipfiles: helper /opt/conda/lib/python3.10/contextlib.py', 1)] 2023-01-11T22:18:44.7354439Z stats [('calls_captured', 3), ('fusions_possible', 2), ('unique_graphs', 1)] 2023-01-11T22:18:44.7354815Z aot_autograd [('total', 1), ('ok', 1)] 2023-01-11T22:18:44.7355122Z ok (0.063s) 2023-01-11T22:18:44.7355282Z 2023-01-11T22:18:44.7355554Z ---------------------------------------------------------------------- 2023-01-11T22:18:44.7355867Z Ran 19 tests in 21.011s 2023-01-11T22:18:44.7356035Z 2023-01-11T22:18:44.7356142Z OK (skipped=9) 2023-01-11T22:18:44.7356306Z 2023-01-11T22:18:44.7356433Z Generating XML reports... 2023-01-11T22:18:44.7357018Z Generated XML report: test-reports/python-unittest/distributed.test_dynamo_distributed/TEST-TestDistributed-20230111221823.xml 2023-01-11T22:18:44.7357793Z Generated XML report: test-reports/python-unittest/distributed.test_dynamo_distributed/TEST-TestDistributedMultiProc-20230111221823.xml 2023-01-11T22:18:44.7358631Z Generated XML report: test-reports/python-unittest/distributed.test_dynamo_distributed/TEST-TestFakeDistributedSingleProc-20230111221823.xml 2023-01-11T22:18:44.7359022Z 2023-01-11T22:18:44.7359387Z ##[endgroup] 2023-01-11T22:18:44.7360057Z FINISHED PRINTING LOG FILE of distributed/test_dynamo_distributed (/var/lib/jenkins/workspace/test/test-reports/distributed-test_dynamo_distributed_wj6tgrhi) 2023-01-11T22:18:44.7360410Z 2023-01-11T22:18:44.7360693Z Running distributed/_tensor/test_redistribute ... [2023-01-11 22:18:44.728614] 2023-01-11T22:18:44.7361385Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_redistribute.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:18:44.728856] 2023-01-11T22:19:14.7491179Z 2023-01-11T22:19:14.7491668Z Expand the folded group to see the log file of distributed/_tensor/test_redistribute 2023-01-11T22:19:14.7495654Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_redistribute (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_redistribute_mxmxhe4s) 2023-01-11T22:19:14.7496021Z 2023-01-11T22:19:14.7498210Z Running tests... 2023-01-11T22:19:14.7498956Z ---------------------------------------------------------------------- 2023-01-11T22:19:14.7499555Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_redistribute 2023-01-11T22:19:14.7500069Z test_multi_dim_mesh (__main__.MultiDimRedistributeTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:19:14.7500538Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59803 2023-01-11T22:19:14.7500990Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59804 2023-01-11T22:19:14.7501453Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 59805 2023-01-11T22:19:14.7501860Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 59806 2023-01-11T22:19:14.7502301Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 59807 2023-01-11T22:19:14.7502736Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 59808 2023-01-11T22:19:14.7503167Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 59809 2023-01-11T22:19:14.7504271Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 59810 2023-01-11T22:19:14.7504945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:19:14.7505398Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:19:14.7505968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:19:14.7506647Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:19:14.7507248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:19:14.7507739Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:19:14.7508290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:19:14.7508730Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:19:14.7509380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:19:14.7509825Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:19:14.7510410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:19:14.7510884Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:19:14.7511465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:19:14.7511880Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:19:14.7512448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:19:14.7513017Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:19:14.7513589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:19:14.7514035Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:19:14.7514595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:19:14.7515041Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:19:14.7515617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:19:14.7516056Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:19:14.7516616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:19:14.7517057Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:19:14.7517626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:19:14.7518060Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:19:14.7518610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:19:14.7519061Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:19:14.7519628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:19:14.7520067Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:19:14.7520609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:19:14.7521070Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:19:14.7521503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2023-01-11T22:19:14.7521972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:19:14.7522420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T22:19:14.7522875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2023-01-11T22:19:14.7523330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:19:14.7523853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T22:19:14.7524311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2023-01-11T22:19:14.7524748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2023-01-11T22:19:14.7525130Z skip: Need at least 8 CUDA devices (4.085s) 2023-01-11T22:19:14.7525622Z test_partial_to_replicate_forward_backward (__main__.RedistributeTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60075 2023-01-11T22:19:14.7526166Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60076 2023-01-11T22:19:14.7526756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:19:14.7527199Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:19:14.7527765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:19:14.7528216Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:19:14.7528787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:19:14.7529230Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:19:14.7529847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:19:14.7530657Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:19:14.7531092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:19:14.7531574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:19:14.7532038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:19:14.7532519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:19:14.7533178Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:19:14.7533863Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:19:14.7534240Z ok (3.910s) 2023-01-11T22:19:14.7534658Z test_partial_to_shard (__main__.RedistributeTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60158 2023-01-11T22:19:14.7535168Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60159 2023-01-11T22:19:14.7535765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:19:14.7536194Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:19:14.7536763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:19:14.7537223Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:19:14.7537776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:19:14.7538213Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:19:14.7538780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:19:14.7539242Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:19:14.7539658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:19:14.7540139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:19:14.7540727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:19:14.7541203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:19:14.7541842Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:19:14.7542527Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:19:14.7542918Z ok (3.811s) 2023-01-11T22:19:14.7543321Z test_replicate_to_partial (__main__.RedistributeTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60237 2023-01-11T22:19:14.7543841Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60238 2023-01-11T22:19:14.7544440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:19:14.7544889Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:19:14.7545441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:19:14.7545907Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:19:14.7546477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:19:14.7546980Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:19:14.7547561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:19:14.7548021Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:19:14.7548450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:19:14.7548916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:19:14.7549400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:19:14.7549878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:19:14.7550528Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:19:14.7551206Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:19:14.7551731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T22:19:14.7552214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T22:19:14.7552860Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:19:14.7553371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2023-01-11T22:19:14.7554007Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:19:14.7554523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2023-01-11T22:19:14.7555148Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2023-01-11T22:19:14.7555826Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2023-01-11T22:19:14.7556389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2023-01-11T22:19:14.7556872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2023-01-11T22:19:14.7557497Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2023-01-11T22:19:14.7558238Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2023-01-11T22:19:14.7558630Z ok (4.111s) 2023-01-11T22:19:14.7559078Z test_replicate_to_replicate_forward_backward (__main__.RedistributeTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60335 2023-01-11T22:19:14.7559608Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60336 2023-01-11T22:19:14.7560209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:19:14.7560654Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:19:14.7561205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:19:14.7561673Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:19:14.7562255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:19:14.7562696Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:19:14.7563240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:19:14.7563703Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:19:14.7564193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:19:14.7564685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:19:14.7565148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:19:14.7565625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:19:14.7566277Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:19:14.7566939Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:19:14.7567331Z ok (3.910s) 2023-01-11T22:19:14.7567771Z test_replicate_to_shard_forward_backward (__main__.RedistributeTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60418 2023-01-11T22:19:14.7568313Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60419 2023-01-11T22:19:14.7568902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:19:14.7569351Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:19:14.7569918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:19:14.7570647Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:19:14.7571220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:19:14.7571662Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:19:14.7572229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:19:14.7572678Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:19:14.7573113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:19:14.7573596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:19:14.7574074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:19:14.7574534Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:19:14.7575278Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:19:14.7575959Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:19:14.7576335Z ok (3.911s) 2023-01-11T22:19:14.7576779Z test_shard_to_replicate_forward_backward (__main__.RedistributeTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60501 2023-01-11T22:19:14.7577319Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60502 2023-01-11T22:19:14.7577923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:19:14.7578350Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:19:14.7578913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:19:14.7579382Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:19:14.7579956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:19:14.7580378Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:19:14.7581015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:19:14.7581483Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:19:14.7581899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:19:14.7582388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:19:14.7582866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:19:14.7583351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:19:14.7583982Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:19:14.7584661Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:19:14.7585054Z ok (4.011s) 2023-01-11T22:19:14.7585202Z 2023-01-11T22:19:14.7585492Z ---------------------------------------------------------------------- 2023-01-11T22:19:14.7585802Z Ran 7 tests in 27.750s 2023-01-11T22:19:14.7585963Z 2023-01-11T22:19:14.7586071Z OK (skipped=1) 2023-01-11T22:19:14.7586223Z 2023-01-11T22:19:14.7586348Z Generating XML reports... 2023-01-11T22:19:14.7586917Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_redistribute/TEST-RedistributeTest-20230111221846.xml 2023-01-11T22:19:14.7587704Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_redistribute/TEST-MultiDimRedistributeTest-20230111221846.xml 2023-01-11T22:19:14.7588076Z 2023-01-11T22:19:14.7588434Z ##[endgroup] 2023-01-11T22:19:14.7589020Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_redistribute (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_redistribute_mxmxhe4s) 2023-01-11T22:19:14.7589384Z 2023-01-11T22:19:14.7589646Z Running distributed/test_c10d_spawn_ucc ... [2023-01-11 22:19:14.749250] 2023-01-11T22:19:14.7590346Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_spawn_ucc.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:19:14.749501] 2023-01-11T22:20:10.8106788Z 2023-01-11T22:20:10.8107302Z Expand the folded group to see the log file of distributed/test_c10d_spawn_ucc 2023-01-11T22:20:10.8110890Z ##[group]PRINTING LOG FILE of distributed/test_c10d_spawn_ucc (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_spawn_ucc_3sr61zhe) 2023-01-11T22:20:10.8111756Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqoi03e3c 2023-01-11T22:20:10.8112302Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqoi03e3c/_remote_module_non_scriptable.py 2023-01-11T22:20:10.8112728Z INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:20:10.8113048Z 2023-01-11T22:20:10.8113356Z 2023-01-11T22:20:10.8114315Z , <__main__.TestDistributedNNFunctionsUcc testMethod=test_all_to_all>, <__main__.TestDistributedNNFunctionsUcc testMethod=test_all_to_all_single>, <__main__.TestDistributedNNFunctionsUcc testMethod=test_allreduce>, <__main__.TestDistributedNNFunctionsUcc testMethod=test_broadcast>, <__main__.TestDistributedNNFunctionsUcc testMethod=test_reduce>]> 2023-01-11T22:20:10.8115278Z test_all_gather (__main__.TestDistributedNNFunctionsUcc) 2023-01-11T22:20:10.8115684Z test_all_to_all (__main__.TestDistributedNNFunctionsUcc) 2023-01-11T22:20:10.8116063Z test_all_to_all_single (__main__.TestDistributedNNFunctionsUcc) 2023-01-11T22:20:10.8116469Z test_allreduce (__main__.TestDistributedNNFunctionsUcc) 2023-01-11T22:20:10.8116859Z test_broadcast (__main__.TestDistributedNNFunctionsUcc) 2023-01-11T22:20:10.8117225Z test_reduce (__main__.TestDistributedNNFunctionsUcc) 2023-01-11T22:20:10.8118037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:20:10.8118521Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:20:10.8119111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:20:10.8119575Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:20:10.8120036Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj_u55teq 2023-01-11T22:20:10.8120581Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj_u55teq/_remote_module_non_scriptable.py 2023-01-11T22:20:10.8121011Z INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:20:10.8121195Z 2023-01-11T22:20:10.8121307Z Running tests... 2023-01-11T22:20:10.8121724Z ---------------------------------------------------------------------- 2023-01-11T22:20:10.8122264Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc 2023-01-11T22:20:10.8122818Z test_all_gather (__main__.TestDistributedNNFunctionsUcc) ... skip: runs into illegal memory access on first assertEqual check when run locally (0.000s) 2023-01-11T22:20:10.8123166Z 2023-01-11T22:20:10.8123432Z ---------------------------------------------------------------------- 2023-01-11T22:20:10.8123772Z Ran 1 test in 0.001s 2023-01-11T22:20:10.8123945Z 2023-01-11T22:20:10.8124049Z OK (skipped=1) 2023-01-11T22:20:10.8124201Z 2023-01-11T22:20:10.8124310Z Generating XML reports... 2023-01-11T22:20:10.8124949Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20230111221921.xml 2023-01-11T22:20:10.8125679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:20:10.8126128Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:20:10.8126686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:20:10.8127159Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:20:10.8127628Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp96lm5ojm 2023-01-11T22:20:10.8128152Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp96lm5ojm/_remote_module_non_scriptable.py 2023-01-11T22:20:10.8128591Z INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:20:10.8128871Z 2023-01-11T22:20:10.8129039Z Running tests... 2023-01-11T22:20:10.8129458Z ---------------------------------------------------------------------- 2023-01-11T22:20:10.8130006Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc 2023-01-11T22:20:10.8131009Z test_all_to_all (__main__.TestDistributedNNFunctionsUcc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60689 2023-01-11T22:20:10.8131530Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60690 2023-01-11T22:20:10.8132142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:20:10.8132596Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:20:10.8133148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:20:10.8133618Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:20:10.8134196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:20:10.8134637Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:20:10.8135185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:20:10.8135648Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:20:10.8136200Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpazb9jm5_ 2023-01-11T22:20:10.8136753Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpazb9jm5_/_remote_module_non_scriptable.py 2023-01-11T22:20:10.8137268Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpacna92wh 2023-01-11T22:20:10.8137799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpacna92wh/_remote_module_non_scriptable.py 2023-01-11T22:20:10.8138230Z INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:20:10.8138614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:20:10.8139100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:20:10.8139502Z INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:20:10.8139898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:20:10.8140365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:20:10.8141025Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:20:10.8141714Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:20:10.8142256Z [1673475570.264801] [76a1849f93d4:60689:0] ec_cuda.c:343 cuda ec WARN CUDA cooperative groups are not supported. Fall back to non cooperative launch. 2023-01-11T22:20:10.8142764Z [1673475570.289726] [76a1849f93d4:60689:0] parser.c:1993 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-01-11T22:20:10.8143230Z [1673475570.289726] [76a1849f93d4:60689:0] parser.c:1993 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-01-11T22:20:10.8143735Z [1673475570.271953] [76a1849f93d4:60690:0] ec_cuda.c:343 cuda ec WARN CUDA cooperative groups are not supported. Fall back to non cooperative launch. 2023-01-11T22:20:10.8144223Z [1673475570.290368] [76a1849f93d4:60690:0] parser.c:1993 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-01-11T22:20:10.8144668Z [1673475570.290368] [76a1849f93d4:60690:0] parser.c:1993 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-01-11T22:20:10.8145006Z ok (5.727s) 2023-01-11T22:20:10.8145154Z 2023-01-11T22:20:10.8145423Z ---------------------------------------------------------------------- 2023-01-11T22:20:10.8145812Z Ran 1 test in 5.727s 2023-01-11T22:20:10.8145975Z 2023-01-11T22:20:10.8146066Z OK 2023-01-11T22:20:10.8146202Z 2023-01-11T22:20:10.8146324Z Generating XML reports... 2023-01-11T22:20:10.8146958Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20230111221925.xml 2023-01-11T22:20:10.8147688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:20:10.8148142Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:20:10.8148718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:20:10.8149169Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:20:10.8149630Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf3ti1g2d 2023-01-11T22:20:10.8150169Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf3ti1g2d/_remote_module_non_scriptable.py 2023-01-11T22:20:10.8150577Z INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:20:10.8150774Z 2023-01-11T22:20:10.8150883Z Running tests... 2023-01-11T22:20:10.8151288Z ---------------------------------------------------------------------- 2023-01-11T22:20:10.8151823Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc 2023-01-11T22:20:10.8152434Z test_all_to_all_single (__main__.TestDistributedNNFunctionsUcc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60810 2023-01-11T22:20:10.8152992Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60811 2023-01-11T22:20:10.8153598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:20:10.8154027Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:20:10.8154607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:20:10.8155075Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:20:10.8155649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:20:10.8156075Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:20:10.8156648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:20:10.8157109Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:20:10.8157571Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsrme07g9 2023-01-11T22:20:10.8158091Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsrme07g9/_remote_module_non_scriptable.py 2023-01-11T22:20:10.8158671Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe2qfd1gh 2023-01-11T22:20:10.8159207Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe2qfd1gh/_remote_module_non_scriptable.py 2023-01-11T22:20:10.8159616Z INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:20:10.8171147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:20:10.8171695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:20:10.8172106Z INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:20:10.8172490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:20:10.8172971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:20:10.8173666Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:20:10.8174505Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:20:10.8175047Z [1673475579.858502] [76a1849f93d4:60810:0] ec_cuda.c:343 cuda ec WARN CUDA cooperative groups are not supported. Fall back to non cooperative launch. 2023-01-11T22:20:10.8175548Z [1673475579.877270] [76a1849f93d4:60810:0] parser.c:1993 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-01-11T22:20:10.8176017Z [1673475579.877270] [76a1849f93d4:60810:0] parser.c:1993 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-01-11T22:20:10.8176519Z [1673475579.863655] [76a1849f93d4:60811:0] ec_cuda.c:343 cuda ec WARN CUDA cooperative groups are not supported. Fall back to non cooperative launch. 2023-01-11T22:20:10.8176993Z [1673475579.881279] [76a1849f93d4:60811:0] parser.c:1993 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-01-11T22:20:10.8177458Z [1673475579.881279] [76a1849f93d4:60811:0] parser.c:1993 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-01-11T22:20:10.8177795Z ok (5.727s) 2023-01-11T22:20:10.8177941Z 2023-01-11T22:20:10.8178214Z ---------------------------------------------------------------------- 2023-01-11T22:20:10.8178522Z Ran 1 test in 5.727s 2023-01-11T22:20:10.8178682Z 2023-01-11T22:20:10.8178775Z OK 2023-01-11T22:20:10.8178907Z 2023-01-11T22:20:10.8179029Z Generating XML reports... 2023-01-11T22:20:10.8179728Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20230111221935.xml 2023-01-11T22:20:10.8180477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:20:10.8180926Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:20:10.8181498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:20:10.8181953Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:20:10.8182413Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdpjqe46f 2023-01-11T22:20:10.8182952Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdpjqe46f/_remote_module_non_scriptable.py 2023-01-11T22:20:10.8183363Z INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:20:10.8183557Z 2023-01-11T22:20:10.8183663Z Running tests... 2023-01-11T22:20:10.8184072Z ---------------------------------------------------------------------- 2023-01-11T22:20:10.8184603Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc 2023-01-11T22:20:10.8185150Z test_allreduce (__main__.TestDistributedNNFunctionsUcc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60931 2023-01-11T22:20:10.8185684Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60932 2023-01-11T22:20:10.8186294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:20:10.8186739Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:20:10.8187292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:20:10.8187754Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:20:10.8188327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:20:10.8188751Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:20:10.8189318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:20:10.8189776Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:20:10.8190232Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnc9zcm_b 2023-01-11T22:20:10.8190826Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnc9zcm_b/_remote_module_non_scriptable.py 2023-01-11T22:20:10.8191359Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9kz5b2qi 2023-01-11T22:20:10.8191890Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9kz5b2qi/_remote_module_non_scriptable.py 2023-01-11T22:20:10.8192297Z INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:20:10.8192703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:20:10.8193191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:20:10.8193590Z INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:20:10.8193967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:20:10.8194446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:20:10.8195110Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:20:10.8195776Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:20:10.8196387Z [1673475589.625246] [76a1849f93d4:60931:0] ec_cuda.c:343 cuda ec WARN CUDA cooperative groups are not supported. Fall back to non cooperative launch. 2023-01-11T22:20:10.8196893Z [1673475589.643694] [76a1849f93d4:60931:0] parser.c:1993 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-01-11T22:20:10.8197358Z [1673475589.643694] [76a1849f93d4:60931:0] parser.c:1993 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-01-11T22:20:10.8197862Z [1673475589.633133] [76a1849f93d4:60932:0] ec_cuda.c:343 cuda ec WARN CUDA cooperative groups are not supported. Fall back to non cooperative launch. 2023-01-11T22:20:10.8198342Z [1673475589.651390] [76a1849f93d4:60932:0] parser.c:1993 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-01-11T22:20:10.8198799Z [1673475589.651390] [76a1849f93d4:60932:0] parser.c:1993 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-01-11T22:20:10.8199134Z ok (5.828s) 2023-01-11T22:20:10.8199282Z 2023-01-11T22:20:10.8199541Z ---------------------------------------------------------------------- 2023-01-11T22:20:10.8199864Z Ran 1 test in 5.828s 2023-01-11T22:20:10.8200023Z 2023-01-11T22:20:10.8200115Z OK 2023-01-11T22:20:10.8200246Z 2023-01-11T22:20:10.8200368Z Generating XML reports... 2023-01-11T22:20:10.8200981Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20230111221944.xml 2023-01-11T22:20:10.8201709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:20:10.8202163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:20:10.8202714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:20:10.8203181Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:20:10.8203639Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyj2fv8oj 2023-01-11T22:20:10.8204175Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyj2fv8oj/_remote_module_non_scriptable.py 2023-01-11T22:20:10.8204583Z INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:20:10.8204777Z 2023-01-11T22:20:10.8204883Z Running tests... 2023-01-11T22:20:10.8205285Z ---------------------------------------------------------------------- 2023-01-11T22:20:10.8205796Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc 2023-01-11T22:20:10.8206359Z test_broadcast (__main__.TestDistributedNNFunctionsUcc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61050 2023-01-11T22:20:10.8206954Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61051 2023-01-11T22:20:10.8207562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:20:10.8207988Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:20:10.8208552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:20:10.8209014Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:20:10.8209583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:20:10.8210002Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:20:10.8210911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:20:10.8211387Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:20:10.8211850Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvxqarbkm 2023-01-11T22:20:10.8212369Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvxqarbkm/_remote_module_non_scriptable.py 2023-01-11T22:20:10.8213000Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpboth8hgm 2023-01-11T22:20:10.8213544Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpboth8hgm/_remote_module_non_scriptable.py 2023-01-11T22:20:10.8213967Z INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:20:10.8214349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:20:10.8214831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:20:10.8215235Z INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:20:10.8215616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:20:10.8216097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:20:10.8216755Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:20:10.8217445Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:20:10.8217988Z [1673475599.304723] [76a1849f93d4:61050:0] ec_cuda.c:343 cuda ec WARN CUDA cooperative groups are not supported. Fall back to non cooperative launch. 2023-01-11T22:20:10.8218488Z [1673475599.323351] [76a1849f93d4:61050:0] parser.c:1993 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-01-11T22:20:10.8218951Z [1673475599.323351] [76a1849f93d4:61050:0] parser.c:1993 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-01-11T22:20:10.8219456Z [1673475599.312438] [76a1849f93d4:61051:0] ec_cuda.c:343 cuda ec WARN CUDA cooperative groups are not supported. Fall back to non cooperative launch. 2023-01-11T22:20:10.8219928Z [1673475599.331017] [76a1849f93d4:61051:0] parser.c:1993 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-01-11T22:20:10.8220389Z [1673475599.331017] [76a1849f93d4:61051:0] parser.c:1993 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-01-11T22:20:10.8220725Z ok (5.728s) 2023-01-11T22:20:10.8220872Z 2023-01-11T22:20:10.8221142Z ---------------------------------------------------------------------- 2023-01-11T22:20:10.8221452Z Ran 1 test in 5.728s 2023-01-11T22:20:10.8221614Z 2023-01-11T22:20:10.8221705Z OK 2023-01-11T22:20:10.8221836Z 2023-01-11T22:20:10.8221957Z Generating XML reports... 2023-01-11T22:20:10.8222571Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20230111221954.xml 2023-01-11T22:20:10.8223382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:20:10.8223830Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:20:10.8224397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:20:10.8224850Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:20:10.8225309Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ikccoar 2023-01-11T22:20:10.8225845Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ikccoar/_remote_module_non_scriptable.py 2023-01-11T22:20:10.8226269Z INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:20:10.8226447Z 2023-01-11T22:20:10.8226554Z Running tests... 2023-01-11T22:20:10.8226952Z ---------------------------------------------------------------------- 2023-01-11T22:20:10.8227486Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc 2023-01-11T22:20:10.8228029Z test_reduce (__main__.TestDistributedNNFunctionsUcc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61169 2023-01-11T22:20:10.8228559Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61170 2023-01-11T22:20:10.8229212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:20:10.8229666Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:20:10.8230224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:20:10.8230690Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:20:10.8231260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:20:10.8231684Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:20:10.8232244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:20:10.8232700Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:20:10.8233159Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxztgyepw 2023-01-11T22:20:10.8233680Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxztgyepw/_remote_module_non_scriptable.py 2023-01-11T22:20:10.8234210Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphb9k9kar 2023-01-11T22:20:10.8234740Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphb9k9kar/_remote_module_non_scriptable.py 2023-01-11T22:20:10.8235148Z INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:20:10.8235550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:20:10.8236033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:20:10.8236430Z INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:20:10.8236810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:20:10.8237286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:20:10.8237944Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:20:10.8238616Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:20:10.8239173Z [1673475609.007750] [76a1849f93d4:61170:0] ec_cuda.c:343 cuda ec WARN CUDA cooperative groups are not supported. Fall back to non cooperative launch. 2023-01-11T22:20:10.8239734Z [1673475609.025880] [76a1849f93d4:61170:0] parser.c:1993 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-01-11T22:20:10.8240198Z [1673475609.025880] [76a1849f93d4:61170:0] parser.c:1993 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-01-11T22:20:10.8240700Z [1673475608.998431] [76a1849f93d4:61169:0] ec_cuda.c:343 cuda ec WARN CUDA cooperative groups are not supported. Fall back to non cooperative launch. 2023-01-11T22:20:10.8241182Z [1673475609.016876] [76a1849f93d4:61169:0] parser.c:1993 UCX WARN unused environment variables: UCX_COMMIT; UCX_HOME 2023-01-11T22:20:10.8241639Z [1673475609.016876] [76a1849f93d4:61169:0] parser.c:1993 UCX WARN (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning) 2023-01-11T22:20:10.8241974Z ok (5.727s) 2023-01-11T22:20:10.8242121Z 2023-01-11T22:20:10.8242374Z ---------------------------------------------------------------------- 2023-01-11T22:20:10.8242702Z Ran 1 test in 5.727s 2023-01-11T22:20:10.8242862Z 2023-01-11T22:20:10.8242952Z OK 2023-01-11T22:20:10.8243085Z 2023-01-11T22:20:10.8243206Z Generating XML reports... 2023-01-11T22:20:10.8243818Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20230111222004.xml 2023-01-11T22:20:10.8244198Z 2023-01-11T22:20:10.8244555Z ##[endgroup] 2023-01-11T22:20:10.8245180Z FINISHED PRINTING LOG FILE of distributed/test_c10d_spawn_ucc (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_spawn_ucc_3sr61zhe) 2023-01-11T22:20:10.8245527Z 2023-01-11T22:20:10.8245776Z Running distributed/fsdp/test_fsdp_meta ... [2023-01-11 22:20:10.810793] 2023-01-11T22:20:10.8246438Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_meta.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:20:10.811046] 2023-01-11T22:20:49.8006459Z 2023-01-11T22:20:49.8007139Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_meta 2023-01-11T22:20:49.8008062Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_meta (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_meta_2vmv8u9p) 2023-01-11T22:20:49.8011105Z 2023-01-11T22:20:49.8011525Z Running tests... 2023-01-11T22:20:49.8012372Z ---------------------------------------------------------------------- 2023-01-11T22:20:49.8012926Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_meta 2023-01-11T22:20:49.8013442Z test_bad_arg_meta (__main__.TestFSDPWithMetaDevice) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:20:49.8013918Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61288 2023-01-11T22:20:49.8015214Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61289 2023-01-11T22:20:49.8015914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:20:49.8016577Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:20:49.8017135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:20:49.8017603Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:20:49.8018171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:20:49.8018592Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:20:49.8019171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:20:49.8019632Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:20:49.8020096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:20:49.8020569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:20:49.8021692Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:20:49.8022954Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:20:49.8023972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:20:49.8024915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:20:49.8025709Z dist init r=0, world=2 2023-01-11T22:20:49.8026164Z dist init r=1, world=2 2023-01-11T22:20:49.8026440Z ok (5.632s) 2023-01-11T22:20:49.8026911Z test_bad_arg_torchdistx (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s) 2023-01-11T22:20:49.8027436Z test_meta_device_with_mixed_precision (__main__.TestFSDPWithMetaDevice) 2023-01-11T22:20:49.8027943Z Tests meta device initialization with a ``param_init_fn`` when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61367 2023-01-11T22:20:49.8028467Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61368 2023-01-11T22:20:49.8029084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:20:49.8029514Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:20:49.8030223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:20:49.8030702Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:20:49.8031281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:20:49.8031706Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:20:49.8032272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:20:49.8032739Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:20:49.8033169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:20:49.8033657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:20:49.8034310Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:20:49.8034991Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:20:49.8035493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:20:49.8035956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:20:49.8036314Z dist init r=1, world=2 2023-01-11T22:20:49.8036547Z dist init r=0, world=2 2023-01-11T22:20:49.8036783Z ok (3.913s) 2023-01-11T22:20:49.8037265Z test_nested_model_with_meta_device_default_init_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61446 2023-01-11T22:20:49.8037841Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61447 2023-01-11T22:20:49.8038434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:20:49.8038883Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:20:49.8039448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:20:49.8039898Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:20:49.8040466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:20:49.8040981Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:20:49.8041547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:20:49.8041992Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:20:49.8042441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:20:49.8042927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:20:49.8043572Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:20:49.8044237Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:20:49.8044755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:20:49.8045221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:20:49.8045559Z dist init r=0, world=2 2023-01-11T22:20:49.8045808Z dist init r=1, world=2 2023-01-11T22:20:49.8046048Z ok (4.511s) 2023-01-11T22:20:49.8046630Z test_nested_model_with_meta_device_default_init_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61529 2023-01-11T22:20:49.8047211Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61530 2023-01-11T22:20:49.8047817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:20:49.8048266Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:20:49.8048815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:20:49.8049284Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:20:49.8049855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:20:49.8050595Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:20:49.8051160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:20:49.8051630Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:20:49.8052079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:20:49.8052553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:20:49.8053200Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:20:49.8053892Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:20:49.8054407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:20:49.8054854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:20:49.8055209Z dist init r=0, world=2 2023-01-11T22:20:49.8055457Z dist init r=1, world=2 2023-01-11T22:20:49.8055681Z ok (4.511s) 2023-01-11T22:20:49.8056165Z test_nested_model_with_meta_device_reset_params_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61612 2023-01-11T22:20:49.8056746Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61613 2023-01-11T22:20:49.8057353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:20:49.8057881Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:20:49.8058450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:20:49.8058912Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:20:49.8059479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:20:49.8059902Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:20:49.8060509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:20:49.8060967Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:20:49.8061396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:20:49.8061888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:20:49.8062539Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:20:49.8063226Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:20:49.8063724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:20:49.8064260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:20:49.8064625Z dist init r=0, world=2 2023-01-11T22:20:49.8064858Z dist init r=1, world=2 2023-01-11T22:20:49.8065096Z ok (4.511s) 2023-01-11T22:20:49.8065576Z test_nested_model_with_meta_device_reset_params_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61695 2023-01-11T22:20:49.8066158Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61696 2023-01-11T22:20:49.8066753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:20:49.8067200Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:20:49.8067770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:20:49.8068237Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:20:49.8068796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:20:49.8069238Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:20:49.8069801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:20:49.8070248Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:20:49.8070699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:20:49.8071186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:20:49.8071837Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:20:49.8072504Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:20:49.8073018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:20:49.8073483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:20:49.8073817Z dist init r=1, world=2 2023-01-11T22:20:49.8074066Z dist init r=0, world=2 2023-01-11T22:20:49.8074302Z ok (4.512s) 2023-01-11T22:20:49.8074800Z test_nested_model_with_torchdistX_default_init_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.001s) 2023-01-11T22:20:49.8075567Z test_nested_model_with_torchdistX_default_init_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s) 2023-01-11T22:20:49.8076280Z test_nested_model_with_torchdistX_init_fn_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s) 2023-01-11T22:20:49.8076977Z test_nested_model_with_torchdistX_init_fn_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s) 2023-01-11T22:20:49.8077665Z test_simple_model_with_meta_device_default_init (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61778 2023-01-11T22:20:49.8078204Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61779 2023-01-11T22:20:49.8078815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:20:49.8079264Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:20:49.8079831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:20:49.8080278Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:20:49.8080902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:20:49.8081352Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:20:49.8081901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:20:49.8082363Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:20:49.8082814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:20:49.8083297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:20:49.8083932Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:20:49.8084614Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:20:49.8085129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:20:49.8085594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:20:49.8085929Z dist init r=0, world=2 2023-01-11T22:20:49.8086175Z dist init r=1, world=2 2023-01-11T22:20:49.8086411Z ok (4.511s) 2023-01-11T22:20:49.8086861Z test_simple_model_with_meta_device_reset_params (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61861 2023-01-11T22:20:49.8087429Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61862 2023-01-11T22:20:49.8088031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:20:49.8088476Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:20:49.8089031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:20:49.8089519Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:20:49.8090086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:20:49.8090794Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:20:49.8091350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:20:49.8091900Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:20:49.8092345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:20:49.8092822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:20:49.8093479Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:20:49.8094159Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:20:49.8094675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:20:49.8095119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:20:49.8095470Z dist init r=1, world=2 2023-01-11T22:20:49.8095721Z dist init r=0, world=2 2023-01-11T22:20:49.8095942Z ok (4.511s) 2023-01-11T22:20:49.8096416Z test_simple_model_with_torchdistX_default_init (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s) 2023-01-11T22:20:49.8097090Z test_simple_model_with_torchdistX_init_fn (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s) 2023-01-11T22:20:49.8097446Z 2023-01-11T22:20:49.8097784Z ---------------------------------------------------------------------- 2023-01-11T22:20:49.8098106Z Ran 15 tests in 36.618s 2023-01-11T22:20:49.8098270Z 2023-01-11T22:20:49.8098378Z OK (skipped=7) 2023-01-11T22:20:49.8098532Z 2023-01-11T22:20:49.8098654Z Generating XML reports... 2023-01-11T22:20:49.8099243Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_meta/TEST-TestFSDPWithMetaDevice-20230111222012.xml 2023-01-11T22:20:49.8099611Z 2023-01-11T22:20:49.8100069Z ##[endgroup] 2023-01-11T22:20:49.8100657Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_meta (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_meta_2vmv8u9p) 2023-01-11T22:20:49.8101004Z 2023-01-11T22:20:49.8101276Z Running distributed/fsdp/test_fsdp_exec_order ... [2023-01-11 22:20:49.800800] 2023-01-11T22:20:49.8101946Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_exec_order.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:20:49.801075] 2023-01-11T22:21:29.4336941Z 2023-01-11T22:21:29.4337592Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_exec_order 2023-01-11T22:21:29.4338579Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_exec_order (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_exec_order_t7gjne9m) 2023-01-11T22:21:29.4338946Z 2023-01-11T22:21:29.4339060Z Running tests... 2023-01-11T22:21:29.4339571Z ---------------------------------------------------------------------- 2023-01-11T22:21:29.4340151Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_exec_order 2023-01-11T22:21:29.4342462Z test_invalid_first_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestFSDPExecOrder) 2023-01-11T22:21:29.4345854Z Tests that FSDP errors if the all-gather order differs across ranks ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:21:29.4346404Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61979 2023-01-11T22:21:29.4346883Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61980 2023-01-11T22:21:29.4347511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:21:29.4347951Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:21:29.4348518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:21:29.4349238Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:21:29.4349849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:21:29.4350284Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:21:29.4350866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:21:29.4351331Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:21:29.4351819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:21:29.4352323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:21:29.4352974Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:21:29.4353685Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:21:29.4354197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:21:29.4354667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:21:29.4356000Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:21:29.4356889Z warnings.warn( 2023-01-11T22:21:29.4358043Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:21:29.4358838Z warnings.warn( 2023-01-11T22:21:29.4359079Z dist init r=0, world=2 2023-01-11T22:21:29.4359312Z dist init r=1, world=2 2023-01-11T22:21:29.4359547Z ok (6.013s) 2023-01-11T22:21:29.4359929Z test_invalid_first_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestFSDPExecOrder) 2023-01-11T22:21:29.4360629Z Tests that FSDP errors if the all-gather order differs across ranks ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62058 2023-01-11T22:21:29.4361172Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62059 2023-01-11T22:21:29.4361830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:21:29.4362280Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:21:29.4362834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:21:29.4363302Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:21:29.4363882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:21:29.4364319Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:21:29.4364870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:21:29.4365337Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:21:29.4365785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:21:29.4366345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:21:29.4367001Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:21:29.4367683Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:21:29.4368202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:21:29.4368651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:21:29.4369898Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:21:29.4370974Z warnings.warn( 2023-01-11T22:21:29.4372218Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:21:29.4373007Z warnings.warn( 2023-01-11T22:21:29.4373235Z dist init r=1, world=2 2023-01-11T22:21:29.4373485Z dist init r=0, world=2 2023-01-11T22:21:29.4373719Z ok (4.512s) 2023-01-11T22:21:29.4374116Z test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD_iters_before_path_change_1 (__main__.TestFSDPExecOrder) 2023-01-11T22:21:29.4374872Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62137 2023-01-11T22:21:29.4375409Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62138 2023-01-11T22:21:29.4376015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:21:29.4376447Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:21:29.4377016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:21:29.4377486Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:21:29.4378063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:21:29.4378499Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:21:29.4379054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:21:29.4379514Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:21:29.4379943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:21:29.4380441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:21:29.4381091Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:21:29.4381774Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:21:29.4382274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:21:29.4382824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:21:29.4384073Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:21:29.4384858Z warnings.warn( 2023-01-11T22:21:29.4385996Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:21:29.4386780Z warnings.warn( 2023-01-11T22:21:29.4387018Z dist init r=0, world=2 2023-01-11T22:21:29.4387256Z dist init r=1, world=2 2023-01-11T22:21:29.4387493Z ok (4.413s) 2023-01-11T22:21:29.4387889Z test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD_iters_before_path_change_3 (__main__.TestFSDPExecOrder) 2023-01-11T22:21:29.4388690Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62220 2023-01-11T22:21:29.4389242Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62221 2023-01-11T22:21:29.4389827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:21:29.4390273Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:21:29.4390846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:21:29.4391309Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:21:29.4391871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:21:29.4392312Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:21:29.4392881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:21:29.4393398Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:21:29.4393844Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:21:29.4394322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:21:29.4394974Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:21:29.4395664Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:21:29.4396164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:21:29.4396633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:21:29.4397877Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:21:29.4398665Z warnings.warn( 2023-01-11T22:21:29.4399931Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:21:29.4400716Z warnings.warn( 2023-01-11T22:21:29.4400946Z dist init r=0, world=2 2023-01-11T22:21:29.4401195Z dist init r=1, world=2 2023-01-11T22:21:29.4401430Z ok (4.412s) 2023-01-11T22:21:29.4401829Z test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP_iters_before_path_change_1 (__main__.TestFSDPExecOrder) 2023-01-11T22:21:29.4402573Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62303 2023-01-11T22:21:29.4403116Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62304 2023-01-11T22:21:29.4403699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:21:29.4404146Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:21:29.4404714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:21:29.4405245Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:21:29.4405814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:21:29.4406257Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:21:29.4406819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:21:29.4407280Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:21:29.4407707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:21:29.4408197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:21:29.4408843Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:21:29.4409513Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:21:29.4410030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:21:29.4410679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:21:29.4411935Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:21:29.4412717Z warnings.warn( 2023-01-11T22:21:29.4413841Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:21:29.4414623Z warnings.warn( 2023-01-11T22:21:29.4414869Z dist init r=0, world=2 2023-01-11T22:21:29.4415118Z dist init r=1, world=2 2023-01-11T22:21:29.4415428Z ok (4.412s) 2023-01-11T22:21:29.4415846Z test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP_iters_before_path_change_3 (__main__.TestFSDPExecOrder) 2023-01-11T22:21:29.4416598Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62386 2023-01-11T22:21:29.4417118Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62387 2023-01-11T22:21:29.4417723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:21:29.4418168Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:21:29.4418739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:21:29.4419189Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:21:29.4419766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:21:29.4420208Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:21:29.4420774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:21:29.4421218Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:21:29.4421735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:21:29.4422235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:21:29.4422871Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:21:29.4423557Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:21:29.4424078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:21:29.4424540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:21:29.4425776Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:21:29.4426556Z warnings.warn( 2023-01-11T22:21:29.4427697Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:21:29.4428479Z warnings.warn( 2023-01-11T22:21:29.4428725Z dist init r=0, world=2 2023-01-11T22:21:29.4428955Z dist init r=1, world=2 2023-01-11T22:21:29.4429190Z ok (4.512s) 2023-01-11T22:21:29.4429666Z test_train_eval_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestFSDPExecOrder) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62469 2023-01-11T22:21:29.4430220Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62470 2023-01-11T22:21:29.4430826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:21:29.4431272Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:21:29.4431842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:21:29.4432359Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:21:29.4432934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:21:29.4433374Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:21:29.4433941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:21:29.4434387Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:21:29.4434831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:21:29.4435319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:21:29.4435946Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:21:29.4436636Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:21:29.4437154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:21:29.4437617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:21:29.4438910Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:21:29.4439690Z warnings.warn( 2023-01-11T22:21:29.4440843Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:21:29.4441624Z warnings.warn( 2023-01-11T22:21:29.4441872Z dist init r=1, world=2 2023-01-11T22:21:29.4442101Z dist init r=0, world=2 2023-01-11T22:21:29.4442336Z ok (4.512s) 2023-01-11T22:21:29.4442812Z test_train_eval_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestFSDPExecOrder) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62552 2023-01-11T22:21:29.4443366Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62553 2023-01-11T22:21:29.4443975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:21:29.4444420Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:21:29.4444981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:21:29.4445408Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:21:29.4445977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:21:29.4446444Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:21:29.4447025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:21:29.4447475Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:21:29.4447922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:21:29.4448488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:21:29.4449122Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:21:29.4449803Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:21:29.4450555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:21:29.4451019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:21:29.4452260Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:21:29.4453052Z warnings.warn( 2023-01-11T22:21:29.4454259Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:21:29.4455051Z warnings.warn( 2023-01-11T22:21:29.4455298Z dist init r=1, world=2 2023-01-11T22:21:29.4455527Z dist init r=0, world=2 2023-01-11T22:21:29.4455762Z ok (4.514s) 2023-01-11T22:21:29.4455909Z 2023-01-11T22:21:29.4456181Z ---------------------------------------------------------------------- 2023-01-11T22:21:29.4456496Z Ran 8 tests in 37.301s 2023-01-11T22:21:29.4456660Z 2023-01-11T22:21:29.4456747Z OK 2023-01-11T22:21:29.4456881Z 2023-01-11T22:21:29.4457001Z Generating XML reports... 2023-01-11T22:21:29.4457603Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_exec_order/TEST-TestFSDPExecOrder-20230111222051.xml 2023-01-11T22:21:29.4457946Z 2023-01-11T22:21:29.4458337Z ##[endgroup] 2023-01-11T22:21:29.4458943Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_exec_order (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_exec_order_t7gjne9m) 2023-01-11T22:21:29.4459309Z 2023-01-11T22:21:29.4459601Z Running distributed/fsdp/test_fsdp_sharded_grad_scaler ... [2023-01-11 22:21:29.433890] 2023-01-11T22:21:29.4460300Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_sharded_grad_scaler.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:21:29.434138] 2023-01-11T22:22:09.9093131Z 2023-01-11T22:22:09.9094165Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_sharded_grad_scaler 2023-01-11T22:22:09.9095167Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_sharded_grad_scaler (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_sharded_grad_scaler_lsv252ff) 2023-01-11T22:22:09.9095559Z 2023-01-11T22:22:09.9095662Z Running tests... 2023-01-11T22:22:09.9101163Z ---------------------------------------------------------------------- 2023-01-11T22:22:09.9101842Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler 2023-01-11T22:22:09.9102341Z test_grad_scaling (__main__.TestShardGradScaler) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:22:09.9102707Z ok (1.665s) 2023-01-11T22:22:09.9103049Z test_inf_gradients_skip_optim_step (__main__.TestShardGradScaler) ... ok (0.002s) 2023-01-11T22:22:09.9103457Z test_scaling_unscaling_sparse (__main__.TestShardGradScaler) ... ok (0.006s) 2023-01-11T22:22:09.9104371Z test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62685 2023-01-11T22:22:09.9105005Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62686 2023-01-11T22:22:09.9105620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:09.9106097Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:09.9106662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:09.9107140Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:09.9107725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:09.9108163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:09.9108736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:09.9109213Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:09.9109669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:22:09.9110252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:22:09.9110907Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:22:09.9111620Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:22:09.9112093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:22:09.9112583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:22:09.9113056Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:22:09.9113560Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:22:09.9114855Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:22:09.9115724Z warnings.warn( 2023-01-11T22:22:09.9116849Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:22:09.9117655Z warnings.warn( 2023-01-11T22:22:09.9118027Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:22:09.9118517Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:22:09.9118889Z dist init r=0, world=2 2023-01-11T22:22:09.9119118Z dist init r=1, world=2 2023-01-11T22:22:09.9119350Z ok (4.513s) 2023-01-11T22:22:09.9119878Z test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62768 2023-01-11T22:22:09.9120482Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62769 2023-01-11T22:22:09.9121186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:09.9121641Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:09.9122224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:09.9122676Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:09.9123230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:09.9123704Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:09.9124238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:09.9124703Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:09.9125156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:22:09.9125649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:22:09.9126286Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:22:09.9127019Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:22:09.9127552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:22:09.9128026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:22:09.9128486Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:22:09.9128965Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:22:09.9130236Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:22:09.9131343Z warnings.warn( 2023-01-11T22:22:09.9132476Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:22:09.9133253Z warnings.warn( 2023-01-11T22:22:09.9133621Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:22:09.9134101Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:22:09.9134448Z dist init r=1, world=2 2023-01-11T22:22:09.9134700Z dist init r=0, world=2 2023-01-11T22:22:09.9134937Z ok (4.512s) 2023-01-11T22:22:09.9135470Z test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62851 2023-01-11T22:22:09.9136112Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62852 2023-01-11T22:22:09.9136722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:09.9137171Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:09.9137821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:09.9138290Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:09.9138864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:09.9139546Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:09.9140083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:09.9140548Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:09.9140998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:22:09.9141487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:22:09.9142137Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:22:09.9142814Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:22:09.9143350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:22:09.9143919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:22:09.9144408Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:22:09.9144888Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:22:09.9146148Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:22:09.9146938Z warnings.warn( 2023-01-11T22:22:09.9148082Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:22:09.9148840Z warnings.warn( 2023-01-11T22:22:09.9149203Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:22:09.9149677Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:22:09.9150025Z dist init r=1, world=2 2023-01-11T22:22:09.9150271Z dist init r=0, world=2 2023-01-11T22:22:09.9150506Z ok (4.512s) 2023-01-11T22:22:09.9151008Z test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62934 2023-01-11T22:22:09.9151631Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62935 2023-01-11T22:22:09.9152238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:09.9152684Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:09.9153236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:09.9153695Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:09.9154341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:09.9154781Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:09.9155323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:09.9155778Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:09.9156228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:22:09.9156716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:22:09.9157353Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:22:09.9158032Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:22:09.9158555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:22:09.9159004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:22:09.9159468Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:22:09.9159945Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:22:09.9161260Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:22:09.9162053Z warnings.warn( 2023-01-11T22:22:09.9163184Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:22:09.9164010Z warnings.warn( 2023-01-11T22:22:09.9164379Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:22:09.9164857Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:22:09.9165200Z dist init r=1, world=2 2023-01-11T22:22:09.9165446Z dist init r=0, world=2 2023-01-11T22:22:09.9165680Z ok (4.511s) 2023-01-11T22:22:09.9166189Z test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63017 2023-01-11T22:22:09.9166815Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63018 2023-01-11T22:22:09.9167419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:09.9167885Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:09.9168422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:09.9168884Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:09.9169455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:09.9169876Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:09.9170792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:09.9171355Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:09.9171802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:22:09.9172274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:22:09.9172935Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:22:09.9173619Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:22:09.9174131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:22:09.9174576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:22:09.9175044Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:22:09.9175522Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:22:09.9176854Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:22:09.9177636Z warnings.warn( 2023-01-11T22:22:09.9178782Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:22:09.9179559Z warnings.warn( 2023-01-11T22:22:09.9179820Z File "", line 1, in 2023-01-11T22:22:09.9180167Z File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main 2023-01-11T22:22:09.9180533Z exitcode = _main(fd, parent_sentinel) 2023-01-11T22:22:09.9180898Z File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main 2023-01-11T22:22:09.9181281Z return self._bootstrap(parent_sentinel) 2023-01-11T22:22:09.9181625Z File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap 2023-01-11T22:22:09.9181954Z self.run() 2023-01-11T22:22:09.9182279Z File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run 2023-01-11T22:22:09.9182622Z self._target(*self._args, **self._kwargs) 2023-01-11T22:22:09.9183136Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 824, in _run 2023-01-11T22:22:09.9183523Z self.run_test(test_name, pipe) 2023-01-11T22:22:09.9184024Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-01-11T22:22:09.9184412Z getattr(self, test_name)() 2023-01-11T22:22:09.9184918Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 536, in wrapper 2023-01-11T22:22:09.9185285Z fn() 2023-01-11T22:22:09.9185752Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 248, in instantiated_test 2023-01-11T22:22:09.9186144Z test(self, **param_kwargs) 2023-01-11T22:22:09.9186652Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 167, in wrapper 2023-01-11T22:22:09.9187022Z return func(*args, **kwargs) 2023-01-11T22:22:09.9187528Z File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py", line 171, in test_fsdp_ddp_parity_with_grad_scaler 2023-01-11T22:22:09.9187943Z self._test_fsdp_parity( 2023-01-11T22:22:09.9188457Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1024, in _test_fsdp_parity 2023-01-11T22:22:09.9188859Z fsdp_loss = self._train_for_several_steps( 2023-01-11T22:22:09.9189408Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 860, in _train_for_several_steps 2023-01-11T22:22:09.9189803Z output = model(*input) 2023-01-11T22:22:09.9190255Z File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1482, in _call_impl 2023-01-11T22:22:09.9190636Z return forward_call(*args, **kwargs) 2023-01-11T22:22:09.9191168Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 663, in forward 2023-01-11T22:22:09.9191622Z args, kwargs = _root_pre_forward(self, self, args, kwargs) 2023-01-11T22:22:09.9192160Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 483, in _root_pre_forward 2023-01-11T22:22:09.9192550Z _lazy_init(state, module) 2023-01-11T22:22:09.9193047Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 163, in _lazy_init 2023-01-11T22:22:09.9193530Z _share_state_and_init_handle_attrs(state, root_module) 2023-01-11T22:22:09.9194123Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 178, in _share_state_and_init_handle_attrs 2023-01-11T22:22:09.9194554Z handle.init_flat_param_attributes() 2023-01-11T22:22:09.9195054Z File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context 2023-01-11T22:22:09.9195414Z return func(*args, **kwargs) 2023-01-11T22:22:09.9195935Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 775, in init_flat_param_attributes 2023-01-11T22:22:09.9196319Z p_assert( 2023-01-11T22:22:09.9196762Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2023-01-11T22:22:09.9197144Z traceback.print_stack() 2023-01-11T22:22:09.9197425Z File "", line 1, in 2023-01-11T22:22:09.9197769Z File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main 2023-01-11T22:22:09.9198137Z exitcode = _main(fd, parent_sentinel) 2023-01-11T22:22:09.9198498Z File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main 2023-01-11T22:22:09.9198861Z return self._bootstrap(parent_sentinel) 2023-01-11T22:22:09.9199223Z File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap 2023-01-11T22:22:09.9199552Z self.run() 2023-01-11T22:22:09.9199877Z File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run 2023-01-11T22:22:09.9200226Z self._target(*self._args, **self._kwargs) 2023-01-11T22:22:09.9200730Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 824, in _run 2023-01-11T22:22:09.9201112Z self.run_test(test_name, pipe) 2023-01-11T22:22:09.9201612Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-01-11T22:22:09.9201996Z getattr(self, test_name)() 2023-01-11T22:22:09.9202505Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 536, in wrapper 2023-01-11T22:22:09.9202867Z fn() 2023-01-11T22:22:09.9203331Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 248, in instantiated_test 2023-01-11T22:22:09.9203719Z test(self, **param_kwargs) 2023-01-11T22:22:09.9204219Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 167, in wrapper 2023-01-11T22:22:09.9204656Z return func(*args, **kwargs) 2023-01-11T22:22:09.9205090Z File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py", line 171, in test_fsdp_ddp_parity_with_grad_scaler 2023-01-11T22:22:09.9205503Z self._test_fsdp_parity( 2023-01-11T22:22:09.9206016Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1024, in _test_fsdp_parity 2023-01-11T22:22:09.9206417Z fsdp_loss = self._train_for_several_steps( 2023-01-11T22:22:09.9206962Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 860, in _train_for_several_steps 2023-01-11T22:22:09.9207352Z output = model(*input) 2023-01-11T22:22:09.9207803Z File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1482, in _call_impl 2023-01-11T22:22:09.9208180Z return forward_call(*args, **kwargs) 2023-01-11T22:22:09.9208717Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 663, in forward 2023-01-11T22:22:09.9209171Z args, kwargs = _root_pre_forward(self, self, args, kwargs) 2023-01-11T22:22:09.9209707Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 483, in _root_pre_forward 2023-01-11T22:22:09.9210093Z _lazy_init(state, module) 2023-01-11T22:22:09.9210859Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 163, in _lazy_init 2023-01-11T22:22:09.9211352Z _share_state_and_init_handle_attrs(state, root_module) 2023-01-11T22:22:09.9211950Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 178, in _share_state_and_init_handle_attrs 2023-01-11T22:22:09.9212380Z handle.init_flat_param_attributes() 2023-01-11T22:22:09.9212884Z File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context 2023-01-11T22:22:09.9213243Z return func(*args, **kwargs) 2023-01-11T22:22:09.9213776Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 775, in init_flat_param_attributes 2023-01-11T22:22:09.9214157Z p_assert( 2023-01-11T22:22:09.9214603Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2023-01-11T22:22:09.9214976Z traceback.print_stack() 2023-01-11T22:22:09.9215358Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:22:09.9215845Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:22:09.9216300Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:22:09.9216770Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:22:09.9217124Z dist init r=0, world=2 2023-01-11T22:22:09.9217569Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2023-01-11T22:22:09.9218014Z dist init r=1, world=2 2023-01-11T22:22:09.9218688Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2023-01-11T22:22:09.9219125Z ok (4.613s) 2023-01-11T22:22:09.9219626Z test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63100 2023-01-11T22:22:09.9220235Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63101 2023-01-11T22:22:09.9220844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:09.9221275Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:09.9221843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:09.9222401Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:09.9222979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:09.9223402Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:09.9223970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:09.9224427Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:09.9224873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:22:09.9225344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:22:09.9225996Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:22:09.9226684Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:22:09.9227186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:22:09.9227653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:22:09.9228176Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:22:09.9228666Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:22:09.9229914Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:22:09.9230706Z warnings.warn( 2023-01-11T22:22:09.9231849Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:22:09.9232626Z warnings.warn( 2023-01-11T22:22:09.9232888Z File "", line 1, in 2023-01-11T22:22:09.9233239Z File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main 2023-01-11T22:22:09.9233603Z exitcode = _main(fd, parent_sentinel) 2023-01-11T22:22:09.9233965Z File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main 2023-01-11T22:22:09.9234317Z return self._bootstrap(parent_sentinel) 2023-01-11T22:22:09.9234697Z File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap 2023-01-11T22:22:09.9235030Z self.run() 2023-01-11T22:22:09.9235353Z File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run 2023-01-11T22:22:09.9235696Z self._target(*self._args, **self._kwargs) 2023-01-11T22:22:09.9236209Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 824, in _run 2023-01-11T22:22:09.9236597Z self.run_test(test_name, pipe) 2023-01-11T22:22:09.9237099Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-01-11T22:22:09.9237485Z getattr(self, test_name)() 2023-01-11T22:22:09.9237993Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 536, in wrapper 2023-01-11T22:22:09.9238411Z fn() 2023-01-11T22:22:09.9238893Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 248, in instantiated_test 2023-01-11T22:22:09.9239286Z test(self, **param_kwargs) 2023-01-11T22:22:09.9239784Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 167, in wrapper 2023-01-11T22:22:09.9240156Z return func(*args, **kwargs) 2023-01-11T22:22:09.9240593Z File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py", line 171, in test_fsdp_ddp_parity_with_grad_scaler 2023-01-11T22:22:09.9241007Z self._test_fsdp_parity( 2023-01-11T22:22:09.9241267Z File "", line 1, in 2023-01-11T22:22:09.9241781Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1024, in _test_fsdp_parity 2023-01-11T22:22:09.9242200Z fsdp_loss = self._train_for_several_steps( 2023-01-11T22:22:09.9242746Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 860, in _train_for_several_steps 2023-01-11T22:22:09.9243122Z output = model(*input) 2023-01-11T22:22:09.9243474Z File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main 2023-01-11T22:22:09.9243838Z exitcode = _main(fd, parent_sentinel) 2023-01-11T22:22:09.9244311Z File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1482, in _call_impl 2023-01-11T22:22:09.9244747Z return forward_call(*args, **kwargs) 2023-01-11T22:22:09.9245116Z File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main 2023-01-11T22:22:09.9245470Z return self._bootstrap(parent_sentinel) 2023-01-11T22:22:09.9246019Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 663, in forward 2023-01-11T22:22:09.9246468Z args, kwargs = _root_pre_forward(self, self, args, kwargs) 2023-01-11T22:22:09.9246871Z File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap 2023-01-11T22:22:09.9247186Z self.run() 2023-01-11T22:22:09.9247680Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 483, in _root_pre_forward 2023-01-11T22:22:09.9248067Z _lazy_init(state, module) 2023-01-11T22:22:09.9248397Z File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run 2023-01-11T22:22:09.9248758Z self._target(*self._args, **self._kwargs) 2023-01-11T22:22:09.9249277Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 163, in _lazy_init 2023-01-11T22:22:09.9249702Z _share_state_and_init_handle_attrs(state, root_module) 2023-01-11T22:22:09.9250203Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 824, in _run 2023-01-11T22:22:09.9250882Z self.run_test(test_name, pipe) 2023-01-11T22:22:09.9251438Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 178, in _share_state_and_init_handle_attrs 2023-01-11T22:22:09.9251857Z handle.init_flat_param_attributes() 2023-01-11T22:22:09.9252385Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-01-11T22:22:09.9252777Z getattr(self, test_name)() 2023-01-11T22:22:09.9253263Z File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context 2023-01-11T22:22:09.9253625Z return func(*args, **kwargs) 2023-01-11T22:22:09.9254134Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 536, in wrapper 2023-01-11T22:22:09.9254495Z fn() 2023-01-11T22:22:09.9254971Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 775, in init_flat_param_attributes 2023-01-11T22:22:09.9255353Z p_assert( 2023-01-11T22:22:09.9255844Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 248, in instantiated_test 2023-01-11T22:22:09.9256326Z test(self, **param_kwargs) 2023-01-11T22:22:09.9256793Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2023-01-11T22:22:09.9257172Z traceback.print_stack() 2023-01-11T22:22:09.9257682Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 167, in wrapper 2023-01-11T22:22:09.9258059Z return func(*args, **kwargs) 2023-01-11T22:22:09.9258492Z File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py", line 171, in test_fsdp_ddp_parity_with_grad_scaler 2023-01-11T22:22:09.9258905Z self._test_fsdp_parity( 2023-01-11T22:22:09.9259403Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1024, in _test_fsdp_parity 2023-01-11T22:22:09.9259828Z fsdp_loss = self._train_for_several_steps( 2023-01-11T22:22:09.9260376Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 860, in _train_for_several_steps 2023-01-11T22:22:09.9260768Z output = model(*input) 2023-01-11T22:22:09.9261220Z File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1482, in _call_impl 2023-01-11T22:22:09.9261602Z return forward_call(*args, **kwargs) 2023-01-11T22:22:09.9262205Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 663, in forward 2023-01-11T22:22:09.9262648Z args, kwargs = _root_pre_forward(self, self, args, kwargs) 2023-01-11T22:22:09.9263207Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 483, in _root_pre_forward 2023-01-11T22:22:09.9263634Z _lazy_init(state, module) 2023-01-11T22:22:09.9264131Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 163, in _lazy_init 2023-01-11T22:22:09.9264539Z _share_state_and_init_handle_attrs(state, root_module) 2023-01-11T22:22:09.9265119Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 178, in _share_state_and_init_handle_attrs 2023-01-11T22:22:09.9265546Z handle.init_flat_param_attributes() 2023-01-11T22:22:09.9266031Z File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context 2023-01-11T22:22:09.9266406Z return func(*args, **kwargs) 2023-01-11T22:22:09.9266934Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 775, in init_flat_param_attributes 2023-01-11T22:22:09.9267313Z p_assert( 2023-01-11T22:22:09.9267758Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2023-01-11T22:22:09.9268134Z traceback.print_stack() 2023-01-11T22:22:09.9268517Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:22:09.9268988Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:22:09.9269464Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:22:09.9269936Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:22:09.9270287Z dist init r=1, world=2 2023-01-11T22:22:09.9270737Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2023-01-11T22:22:09.9271181Z dist init r=0, world=2 2023-01-11T22:22:09.9271637Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2023-01-11T22:22:09.9272051Z ok (4.612s) 2023-01-11T22:22:09.9272588Z test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63183 2023-01-11T22:22:09.9273292Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63184 2023-01-11T22:22:09.9273902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:09.9274334Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:09.9274905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:09.9275371Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:09.9275941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:09.9276364Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:09.9276927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:09.9277388Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:09.9277818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:22:09.9278308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:22:09.9279015Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:22:09.9279706Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:22:09.9280206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:22:09.9280671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:22:09.9281147Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:22:09.9281630Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:22:09.9282879Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:22:09.9283664Z warnings.warn( 2023-01-11T22:22:09.9284804Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:22:09.9285572Z warnings.warn( 2023-01-11T22:22:09.9285834Z File "", line 1, in 2023-01-11T22:22:09.9286179Z File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main 2023-01-11T22:22:09.9286544Z exitcode = _main(fd, parent_sentinel) 2023-01-11T22:22:09.9286911Z File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main 2023-01-11T22:22:09.9287262Z return self._bootstrap(parent_sentinel) 2023-01-11T22:22:09.9287642Z File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap 2023-01-11T22:22:09.9287972Z self.run() 2023-01-11T22:22:09.9288280Z File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run 2023-01-11T22:22:09.9288640Z self._target(*self._args, **self._kwargs) 2023-01-11T22:22:09.9289149Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 824, in _run 2023-01-11T22:22:09.9289596Z self.run_test(test_name, pipe) 2023-01-11T22:22:09.9290100Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-01-11T22:22:09.9290771Z getattr(self, test_name)() 2023-01-11T22:22:09.9291283Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 536, in wrapper 2023-01-11T22:22:09.9291636Z fn() 2023-01-11T22:22:09.9292118Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 248, in instantiated_test 2023-01-11T22:22:09.9292508Z test(self, **param_kwargs) 2023-01-11T22:22:09.9293013Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 167, in wrapper 2023-01-11T22:22:09.9293382Z return func(*args, **kwargs) 2023-01-11T22:22:09.9293816Z File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py", line 171, in test_fsdp_ddp_parity_with_grad_scaler 2023-01-11T22:22:09.9294229Z self._test_fsdp_parity( 2023-01-11T22:22:09.9294730Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1024, in _test_fsdp_parity 2023-01-11T22:22:09.9295146Z fsdp_loss = self._train_for_several_steps( 2023-01-11T22:22:09.9295774Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 860, in _train_for_several_steps 2023-01-11T22:22:09.9296181Z output = model(*input) 2023-01-11T22:22:09.9296635Z File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1482, in _call_impl 2023-01-11T22:22:09.9297014Z return forward_call(*args, **kwargs) 2023-01-11T22:22:09.9297549Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 663, in forward 2023-01-11T22:22:09.9297982Z args, kwargs = _root_pre_forward(self, self, args, kwargs) 2023-01-11T22:22:09.9298537Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 483, in _root_pre_forward 2023-01-11T22:22:09.9298925Z _lazy_init(state, module) 2023-01-11T22:22:09.9299421Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 163, in _lazy_init 2023-01-11T22:22:09.9299829Z _share_state_and_init_handle_attrs(state, root_module) 2023-01-11T22:22:09.9300412Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 178, in _share_state_and_init_handle_attrs 2023-01-11T22:22:09.9300839Z handle.init_flat_param_attributes() 2023-01-11T22:22:09.9301330Z File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context 2023-01-11T22:22:09.9301707Z return func(*args, **kwargs) 2023-01-11T22:22:09.9302229Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 775, in init_flat_param_attributes 2023-01-11T22:22:09.9302613Z p_assert( 2023-01-11T22:22:09.9303059Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2023-01-11T22:22:09.9303435Z traceback.print_stack() 2023-01-11T22:22:09.9303713Z File "", line 1, in 2023-01-11T22:22:09.9304057Z File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main 2023-01-11T22:22:09.9304425Z exitcode = _main(fd, parent_sentinel) 2023-01-11T22:22:09.9304788Z File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main 2023-01-11T22:22:09.9305136Z return self._bootstrap(parent_sentinel) 2023-01-11T22:22:09.9305511Z File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap 2023-01-11T22:22:09.9305840Z self.run() 2023-01-11T22:22:09.9306150Z File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run 2023-01-11T22:22:09.9306508Z self._target(*self._args, **self._kwargs) 2023-01-11T22:22:09.9307101Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 824, in _run 2023-01-11T22:22:09.9307482Z self.run_test(test_name, pipe) 2023-01-11T22:22:09.9307986Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-01-11T22:22:09.9308379Z getattr(self, test_name)() 2023-01-11T22:22:09.9308891Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 536, in wrapper 2023-01-11T22:22:09.9309238Z fn() 2023-01-11T22:22:09.9309717Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 248, in instantiated_test 2023-01-11T22:22:09.9310109Z test(self, **param_kwargs) 2023-01-11T22:22:09.9310611Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 167, in wrapper 2023-01-11T22:22:09.9310981Z return func(*args, **kwargs) 2023-01-11T22:22:09.9311417Z File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py", line 171, in test_fsdp_ddp_parity_with_grad_scaler 2023-01-11T22:22:09.9311828Z self._test_fsdp_parity( 2023-01-11T22:22:09.9312324Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1024, in _test_fsdp_parity 2023-01-11T22:22:09.9312740Z fsdp_loss = self._train_for_several_steps( 2023-01-11T22:22:09.9313354Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 860, in _train_for_several_steps 2023-01-11T22:22:09.9313752Z output = model(*input) 2023-01-11T22:22:09.9314209Z File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1482, in _call_impl 2023-01-11T22:22:09.9314591Z return forward_call(*args, **kwargs) 2023-01-11T22:22:09.9315125Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 663, in forward 2023-01-11T22:22:09.9315560Z args, kwargs = _root_pre_forward(self, self, args, kwargs) 2023-01-11T22:22:09.9316111Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 483, in _root_pre_forward 2023-01-11T22:22:09.9316499Z _lazy_init(state, module) 2023-01-11T22:22:09.9316995Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 163, in _lazy_init 2023-01-11T22:22:09.9317406Z _share_state_and_init_handle_attrs(state, root_module) 2023-01-11T22:22:09.9317984Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 178, in _share_state_and_init_handle_attrs 2023-01-11T22:22:09.9318411Z handle.init_flat_param_attributes() 2023-01-11T22:22:09.9318898Z File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context 2023-01-11T22:22:09.9319275Z return func(*args, **kwargs) 2023-01-11T22:22:09.9319795Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 775, in init_flat_param_attributes 2023-01-11T22:22:09.9320178Z p_assert( 2023-01-11T22:22:09.9320620Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2023-01-11T22:22:09.9320995Z traceback.print_stack() 2023-01-11T22:22:09.9321379Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:22:09.9321844Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:22:09.9322321Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:22:09.9322791Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:22:09.9323132Z dist init r=0, world=2 2023-01-11T22:22:09.9323592Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2023-01-11T22:22:09.9324102Z dist init r=1, world=2 2023-01-11T22:22:09.9324559Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2023-01-11T22:22:09.9324971Z ok (4.612s) 2023-01-11T22:22:09.9325495Z test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63266 2023-01-11T22:22:09.9326113Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63267 2023-01-11T22:22:09.9326722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:09.9327151Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:09.9327717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:09.9328190Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:09.9328745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:09.9329182Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:09.9329745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:09.9330271Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:09.9330955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:22:09.9331448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:22:09.9332103Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:22:09.9332793Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:22:09.9333294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:22:09.9333757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:22:09.9334230Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:22:09.9334695Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:22:09.9335958Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:22:09.9336749Z warnings.warn( 2023-01-11T22:22:09.9337893Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:22:09.9338667Z warnings.warn( 2023-01-11T22:22:09.9338927Z File "", line 1, in 2023-01-11T22:22:09.9339278Z File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main 2023-01-11T22:22:09.9339644Z exitcode = _main(fd, parent_sentinel) 2023-01-11T22:22:09.9340005Z File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main 2023-01-11T22:22:09.9340456Z return self._bootstrap(parent_sentinel) 2023-01-11T22:22:09.9340837Z File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap 2023-01-11T22:22:09.9341167Z self.run() 2023-01-11T22:22:09.9341477Z File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run 2023-01-11T22:22:09.9341839Z self._target(*self._args, **self._kwargs) 2023-01-11T22:22:09.9342356Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 824, in _run 2023-01-11T22:22:09.9342743Z self.run_test(test_name, pipe) 2023-01-11T22:22:09.9343244Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-01-11T22:22:09.9343632Z getattr(self, test_name)() 2023-01-11T22:22:09.9344139Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 536, in wrapper 2023-01-11T22:22:09.9344488Z fn() 2023-01-11T22:22:09.9344969Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 248, in instantiated_test 2023-01-11T22:22:09.9345361Z test(self, **param_kwargs) 2023-01-11T22:22:09.9345850Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 167, in wrapper 2023-01-11T22:22:09.9346235Z return func(*args, **kwargs) 2023-01-11T22:22:09.9346758Z File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py", line 171, in test_fsdp_ddp_parity_with_grad_scaler 2023-01-11T22:22:09.9347184Z self._test_fsdp_parity( 2023-01-11T22:22:09.9347681Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1024, in _test_fsdp_parity 2023-01-11T22:22:09.9348098Z fsdp_loss = self._train_for_several_steps( 2023-01-11T22:22:09.9348642Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 860, in _train_for_several_steps 2023-01-11T22:22:09.9349023Z output = model(*input) 2023-01-11T22:22:09.9349490Z File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1482, in _call_impl 2023-01-11T22:22:09.9349868Z return forward_call(*args, **kwargs) 2023-01-11T22:22:09.9350401Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 663, in forward 2023-01-11T22:22:09.9350834Z args, kwargs = _root_pre_forward(self, self, args, kwargs) 2023-01-11T22:22:09.9351390Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 483, in _root_pre_forward 2023-01-11T22:22:09.9351779Z _lazy_init(state, module) 2023-01-11T22:22:09.9352260Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 163, in _lazy_init 2023-01-11T22:22:09.9352685Z _share_state_and_init_handle_attrs(state, root_module) 2023-01-11T22:22:09.9353258Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 178, in _share_state_and_init_handle_attrs 2023-01-11T22:22:09.9353690Z handle.init_flat_param_attributes() 2023-01-11T22:22:09.9354175Z File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context 2023-01-11T22:22:09.9354551Z return func(*args, **kwargs) 2023-01-11T22:22:09.9355071Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 775, in init_flat_param_attributes 2023-01-11T22:22:09.9355440Z p_assert( 2023-01-11T22:22:09.9355900Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2023-01-11T22:22:09.9356277Z traceback.print_stack() 2023-01-11T22:22:09.9356559Z File "", line 1, in 2023-01-11T22:22:09.9356906Z File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main 2023-01-11T22:22:09.9357272Z exitcode = _main(fd, parent_sentinel) 2023-01-11T22:22:09.9357698Z File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main 2023-01-11T22:22:09.9358050Z return self._bootstrap(parent_sentinel) 2023-01-11T22:22:09.9358429Z File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap 2023-01-11T22:22:09.9358759Z self.run() 2023-01-11T22:22:09.9359070Z File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run 2023-01-11T22:22:09.9359429Z self._target(*self._args, **self._kwargs) 2023-01-11T22:22:09.9359938Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 824, in _run 2023-01-11T22:22:09.9360322Z self.run_test(test_name, pipe) 2023-01-11T22:22:09.9360824Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-01-11T22:22:09.9361212Z getattr(self, test_name)() 2023-01-11T22:22:09.9361719Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 536, in wrapper 2023-01-11T22:22:09.9362075Z fn() 2023-01-11T22:22:09.9362552Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 248, in instantiated_test 2023-01-11T22:22:09.9362937Z test(self, **param_kwargs) 2023-01-11T22:22:09.9363458Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 167, in wrapper 2023-01-11T22:22:09.9363845Z return func(*args, **kwargs) 2023-01-11T22:22:09.9364330Z File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py", line 171, in test_fsdp_ddp_parity_with_grad_scaler 2023-01-11T22:22:09.9364750Z self._test_fsdp_parity( 2023-01-11T22:22:09.9365250Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1024, in _test_fsdp_parity 2023-01-11T22:22:09.9365665Z fsdp_loss = self._train_for_several_steps( 2023-01-11T22:22:09.9366213Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 860, in _train_for_several_steps 2023-01-11T22:22:09.9366592Z output = model(*input) 2023-01-11T22:22:09.9367059Z File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1482, in _call_impl 2023-01-11T22:22:09.9367434Z return forward_call(*args, **kwargs) 2023-01-11T22:22:09.9367969Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 663, in forward 2023-01-11T22:22:09.9368404Z args, kwargs = _root_pre_forward(self, self, args, kwargs) 2023-01-11T22:22:09.9368960Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 483, in _root_pre_forward 2023-01-11T22:22:09.9369345Z _lazy_init(state, module) 2023-01-11T22:22:09.9369825Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 163, in _lazy_init 2023-01-11T22:22:09.9370249Z _share_state_and_init_handle_attrs(state, root_module) 2023-01-11T22:22:09.9371040Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 178, in _share_state_and_init_handle_attrs 2023-01-11T22:22:09.9371470Z handle.init_flat_param_attributes() 2023-01-11T22:22:09.9371959Z File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context 2023-01-11T22:22:09.9372339Z return func(*args, **kwargs) 2023-01-11T22:22:09.9372864Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 775, in init_flat_param_attributes 2023-01-11T22:22:09.9373228Z p_assert( 2023-01-11T22:22:09.9373687Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2023-01-11T22:22:09.9374063Z traceback.print_stack() 2023-01-11T22:22:09.9374448Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:22:09.9374916Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:22:09.9375481Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:22:09.9375957Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:22:09.9376297Z dist init r=0, world=2 2023-01-11T22:22:09.9376758Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2023-01-11T22:22:09.9377208Z dist init r=1, world=2 2023-01-11T22:22:09.9377664Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2023-01-11T22:22:09.9378076Z ok (4.612s) 2023-01-11T22:22:09.9378225Z 2023-01-11T22:22:09.9378500Z ---------------------------------------------------------------------- 2023-01-11T22:22:09.9378828Z Ran 11 tests in 38.171s 2023-01-11T22:22:09.9378994Z 2023-01-11T22:22:09.9379073Z OK 2023-01-11T22:22:09.9379205Z 2023-01-11T22:22:09.9379323Z Generating XML reports... 2023-01-11T22:22:09.9379940Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardGradScaler-20230111222131.xml 2023-01-11T22:22:09.9380820Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardedGradScalerParityWithDDP-20230111222131.xml 2023-01-11T22:22:09.9381244Z 2023-01-11T22:22:09.9381615Z ##[endgroup] 2023-01-11T22:22:09.9382333Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_sharded_grad_scaler (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_sharded_grad_scaler_lsv252ff) 2023-01-11T22:22:09.9382733Z 2023-01-11T22:22:09.9382995Z Running distributed/_tensor/test_dtensor ... [2023-01-11 22:22:09.909750] 2023-01-11T22:22:09.9383657Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_dtensor.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:22:09.909996] 2023-01-11T22:22:53.6481553Z 2023-01-11T22:22:53.6482444Z Expand the folded group to see the log file of distributed/_tensor/test_dtensor 2023-01-11T22:22:53.6483618Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_dtensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_dtensor_8i2lie68) 2023-01-11T22:22:53.6486409Z 2023-01-11T22:22:53.6486811Z Running tests... 2023-01-11T22:22:53.6487573Z ---------------------------------------------------------------------- 2023-01-11T22:22:53.6488136Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_dtensor 2023-01-11T22:22:53.6488604Z test_device_mesh_nd (__main__.DTensorMeshTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:22:53.6489108Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63384 2023-01-11T22:22:53.6489923Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63385 2023-01-11T22:22:53.6490740Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63386 2023-01-11T22:22:53.6491201Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63387 2023-01-11T22:22:53.6491663Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 63388 2023-01-11T22:22:53.6492082Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 63389 2023-01-11T22:22:53.6492506Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 63390 2023-01-11T22:22:53.6492951Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 63391 2023-01-11T22:22:53.6493590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6494027Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6494619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6495337Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6495924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6496366Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6496957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6497396Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6497973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6498395Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6498985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6499481Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6500042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6500460Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6501027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6501664Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6502253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6502699Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6503250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6503867Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6504951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6505946Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6506732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6510660Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6511959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6512721Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6513900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6514791Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6516040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6516896Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6518049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6518830Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6519648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:22:53.6520477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T22:22:53.6521488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2023-01-11T22:22:53.6521971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2023-01-11T22:22:53.6522435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2023-01-11T22:22:53.6523011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:22:53.6523467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T22:22:53.6523925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2023-01-11T22:22:53.6524300Z skip: Need at least 8 CUDA devices (4.146s) 2023-01-11T22:22:53.6524762Z test_dtensor_2d_mesh (__main__.DTensorMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63656 2023-01-11T22:22:53.6525279Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63657 2023-01-11T22:22:53.6525725Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63658 2023-01-11T22:22:53.6541607Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63659 2023-01-11T22:22:53.6542105Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 63660 2023-01-11T22:22:53.6542574Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 63661 2023-01-11T22:22:53.6543005Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 63662 2023-01-11T22:22:53.6543452Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 63663 2023-01-11T22:22:53.6544128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6544712Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6545316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6545774Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6546343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6546796Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6547350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6547822Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6548432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6548831Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6549396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6549859Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6550428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6550846Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6551410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6551870Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6552415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6552853Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6553410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6553868Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6554412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6554854Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6555510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6555972Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6556517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6556952Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6557512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6557952Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6558517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6558951Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6559512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6559955Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6560384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2023-01-11T22:22:53.6560849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T22:22:53.6561353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2023-01-11T22:22:53.6561830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2023-01-11T22:22:53.6562286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:22:53.6562747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2023-01-11T22:22:53.6563185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:22:53.6563644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T22:22:53.6564030Z skip: Need at least 8 CUDA devices (2.514s) 2023-01-11T22:22:53.6564503Z test_dtensor_api_device_mesh_context_manager (__main__.DTensorMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63928 2023-01-11T22:22:53.6565045Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63929 2023-01-11T22:22:53.6565540Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63930 2023-01-11T22:22:53.6565989Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63931 2023-01-11T22:22:53.6566413Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 63932 2023-01-11T22:22:53.6566850Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 63933 2023-01-11T22:22:53.6567288Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 63934 2023-01-11T22:22:53.6567730Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 63935 2023-01-11T22:22:53.6568324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6568775Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6569344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6569796Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6570931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6571401Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6571978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6572528Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6573101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6573546Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6574095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6574560Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6575131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6575570Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6576115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6576574Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6577146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6577588Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6578134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6578595Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6579229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6579663Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6580227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6580686Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6581258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6581676Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6582237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6582690Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6583243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6583680Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6584236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6584688Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6585097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T22:22:53.6585572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2023-01-11T22:22:53.6586035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2023-01-11T22:22:53.6586495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2023-01-11T22:22:53.6586941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2023-01-11T22:22:53.6587396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:22:53.6587854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:22:53.6588295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T22:22:53.6588677Z skip: Need at least 8 CUDA devices (2.514s) 2023-01-11T22:22:53.6589159Z test_dtensor_device_mesh_device_conversion (__main__.DTensorMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64200 2023-01-11T22:22:53.6589767Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64201 2023-01-11T22:22:53.6590196Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 64202 2023-01-11T22:22:53.6590632Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 64203 2023-01-11T22:22:53.6591074Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 64204 2023-01-11T22:22:53.6591493Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 64205 2023-01-11T22:22:53.6591924Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 64206 2023-01-11T22:22:53.6592363Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 64207 2023-01-11T22:22:53.6592971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6593409Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6593976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6594441Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6594991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6595494Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6596066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6596530Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6597077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6597519Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6598086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6598543Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6599089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6599530Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6600091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6600529Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6601097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6601535Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6602099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6602538Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6603105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6603540Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6604083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6604539Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6605105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6605615Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6606173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6606697Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6607249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6607711Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6608292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6608756Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6609167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2023-01-11T22:22:53.6609633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:22:53.6610094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2023-01-11T22:22:53.6611121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2023-01-11T22:22:53.6611583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T22:22:53.6612043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:22:53.6612503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T22:22:53.6613032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2023-01-11T22:22:53.6613429Z skip: Need at least 8 CUDA devices (2.514s) 2023-01-11T22:22:53.6613910Z test_dtensor_spec_local_shard_offset (__main__.DTensorMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64472 2023-01-11T22:22:53.6614422Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64473 2023-01-11T22:22:53.6614875Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 64474 2023-01-11T22:22:53.6615314Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 64475 2023-01-11T22:22:53.6615756Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 64476 2023-01-11T22:22:53.6616177Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 64477 2023-01-11T22:22:53.6616611Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 64478 2023-01-11T22:22:53.6617053Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 64479 2023-01-11T22:22:53.6617650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6618099Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6618667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6619130Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6619683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6620126Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6620687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6621150Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6621702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6622142Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6622702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6623140Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6623782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6624223Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6624785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6625229Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6625793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6626235Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6626782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6627241Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6627808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6628246Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6628786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6629248Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6629863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6630310Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6630852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6631309Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6631877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6632305Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6632863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6633316Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6633747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:22:53.6634196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2023-01-11T22:22:53.6634657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:22:53.6635120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2023-01-11T22:22:53.6635560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2023-01-11T22:22:53.6636025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T22:22:53.6636475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T22:22:53.6636925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2023-01-11T22:22:53.6637293Z skip: Need at least 8 CUDA devices (2.514s) 2023-01-11T22:22:53.6637749Z test_dtensor_constructor (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64744 2023-01-11T22:22:53.6638253Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64745 2023-01-11T22:22:53.6638838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6639280Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6639847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6640375Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6640931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6641373Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6641937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6642393Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6642806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:22:53.6643291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:22:53.6643767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:22:53.6644229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:22:53.6644879Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:22:53.6645567Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:22:53.6645961Z ok (3.911s) 2023-01-11T22:22:53.6646410Z test_dtensor_properties (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64823 2023-01-11T22:22:53.6646929Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64824 2023-01-11T22:22:53.6647534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6647981Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6648535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6648996Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6649565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6649987Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6651078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6651547Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6651983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:22:53.6652451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:22:53.6652932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:22:53.6653411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:22:53.6654045Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:22:53.6654732Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:22:53.6655124Z ok (3.911s) 2023-01-11T22:22:53.6655549Z test_dtensor_spec_read_only_after_set (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64902 2023-01-11T22:22:53.6656054Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64903 2023-01-11T22:22:53.6656654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6657094Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6657758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6658207Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6658777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6659225Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6659771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6660232Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6660666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:22:53.6661149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:22:53.6661609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:22:53.6662082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:22:53.6662745Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:22:53.6663482Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:22:53.6663869Z ok (3.910s) 2023-01-11T22:22:53.6664268Z test_dtensor_stride (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64981 2023-01-11T22:22:53.6664768Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64982 2023-01-11T22:22:53.6665403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6665855Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6666424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6666891Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6667444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6667887Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6668449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6668894Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6669323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:22:53.6669802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:22:53.6670279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:22:53.6670733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:22:53.6671383Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:22:53.6672065Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:22:53.6672463Z ok (3.911s) 2023-01-11T22:22:53.6672835Z test_from_local (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65060 2023-01-11T22:22:53.6673322Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65061 2023-01-11T22:22:53.6673933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6674411Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6674982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6675447Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6676037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6676440Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6677003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6677457Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6677870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:22:53.6678350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:22:53.6678829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:22:53.6679303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:22:53.6679930Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:22:53.6680661Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:22:53.6681063Z ok (3.811s) 2023-01-11T22:22:53.6681478Z test_from_local_then_to_local (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65143 2023-01-11T22:22:53.6681969Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65144 2023-01-11T22:22:53.6682569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6683018Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6683569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6684031Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6684591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6685032Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6685577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6686037Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6686467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:22:53.6686936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:22:53.6687403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:22:53.6687872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:22:53.6688511Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:22:53.6689166Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:22:53.6689540Z ok (3.810s) 2023-01-11T22:22:53.6689919Z test_to_local (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65226 2023-01-11T22:22:53.6690910Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65227 2023-01-11T22:22:53.6691627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6692077Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6692643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6693090Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6693662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:22:53.6694102Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:22:53.6694661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:22:53.6695105Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:22:53.6695538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:22:53.6696025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:22:53.6696500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:22:53.6696959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:22:53.6697677Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:22:53.6698376Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:22:53.6698756Z ok (3.911s) 2023-01-11T22:22:53.6698908Z 2023-01-11T22:22:53.6699173Z ---------------------------------------------------------------------- 2023-01-11T22:22:53.6699499Z Ran 12 tests in 41.377s 2023-01-11T22:22:53.6699662Z 2023-01-11T22:22:53.6699768Z OK (skipped=5) 2023-01-11T22:22:53.6699907Z 2023-01-11T22:22:53.6700030Z Generating XML reports... 2023-01-11T22:22:53.6700590Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_dtensor/TEST-DTensorTest-20230111222211.xml 2023-01-11T22:22:53.6701305Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_dtensor/TEST-DTensorMeshTest-20230111222211.xml 2023-01-11T22:22:53.6701636Z 2023-01-11T22:22:53.6702090Z ##[endgroup] 2023-01-11T22:22:53.6702679Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_dtensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_dtensor_8i2lie68) 2023-01-11T22:22:53.6703029Z 2023-01-11T22:22:53.6703296Z Running distributed/algorithms/test_join ... [2023-01-11 22:22:53.648517] 2023-01-11T22:22:53.6703963Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/test_join.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:22:53.648778] 2023-01-11T22:23:41.7250407Z 2023-01-11T22:23:41.7251437Z Expand the folded group to see the log file of distributed/algorithms/test_join 2023-01-11T22:23:41.7252385Z ##[group]PRINTING LOG FILE of distributed/algorithms/test_join (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-test_join_y0rvk38w) 2023-01-11T22:23:41.7252756Z 2023-01-11T22:23:41.7252861Z Running tests... 2023-01-11T22:23:41.7253765Z ---------------------------------------------------------------------- 2023-01-11T22:23:41.7254746Z Test results will be stored in test-reports/python-unittest/distributed.algorithms.test_join 2023-01-11T22:23:41.7255616Z test_join_kwargs (__main__.TestJoin) 2023-01-11T22:23:41.7266984Z Tests passing keyword arguments to the context manager. ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:23:41.7267518Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65344 2023-01-11T22:23:41.7267995Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65345 2023-01-11T22:23:41.7268671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:23:41.7269357Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:23:41.7269956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:23:41.7270419Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:23:41.7271002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:23:41.7271431Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:23:41.7271997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:23:41.7272475Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:23:41.7272952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:23:41.7273391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:23:41.7273876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:23:41.7274374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:23:41.7275088Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:23:41.7275814Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:23:41.7276191Z ok (6.502s) 2023-01-11T22:23:41.7276483Z test_multiple_joinable_disable (__main__.TestJoin) 2023-01-11T22:23:41.7276948Z Tests ``enable=False`` for multiple :class:`Joinable` s. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65424 2023-01-11T22:23:41.7277466Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65425 2023-01-11T22:23:41.7278095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:23:41.7278617Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:23:41.7279181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:23:41.7279647Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:23:41.7280203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:23:41.7280651Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:23:41.7281226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:23:41.7281677Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:23:41.7282116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:23:41.7282601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:23:41.7283082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:23:41.7283542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:23:41.7284197Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:23:41.7284882Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:23:41.7285273Z ok (4.912s) 2023-01-11T22:23:41.7285533Z test_multiple_joinables (__main__.TestJoin) 2023-01-11T22:23:41.7286149Z Tests the main hooks and post-hooks of multiple :class:`Joinable` s ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65504 2023-01-11T22:23:41.7286751Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65505 2023-01-11T22:23:41.7287337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:23:41.7287786Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:23:41.7288360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:23:41.7288829Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:23:41.7289384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:23:41.7289824Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:23:41.7290695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:23:41.7291160Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:23:41.7291591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:23:41.7292076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:23:41.7292631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:23:41.7293102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:23:41.7293760Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:23:41.7294442Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:23:41.7294834Z ok (4.912s) 2023-01-11T22:23:41.7295107Z test_multiple_joinables_throw (__main__.TestJoin) 2023-01-11T22:23:41.7295578Z Tests ``throw_on_early_termination=True`` for multiple ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65584 2023-01-11T22:23:41.7296091Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65585 2023-01-11T22:23:41.7296680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:23:41.7297128Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:23:41.7297694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:23:41.7298162Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:23:41.7298714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:23:41.7299161Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:23:41.7299725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:23:41.7300169Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:23:41.7300601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:23:41.7301091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:23:41.7301570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:23:41.7302029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:23:41.7302676Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:23:41.7303359Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:23:41.7303930Z ok (4.912s) 2023-01-11T22:23:41.7304180Z test_single_joinable (__main__.TestJoin) 2023-01-11T22:23:41.7304782Z Tests the main hooks and post-hooks of a single :class:`Joinable` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65664 2023-01-11T22:23:41.7305312Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65665 2023-01-11T22:23:41.7305900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:23:41.7306346Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:23:41.7306913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:23:41.7307379Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:23:41.7307936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:23:41.7308378Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:23:41.7308941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:23:41.7309387Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:23:41.7309872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:23:41.7310362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:23:41.7310840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:23:41.7311307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:23:41.7311956Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:23:41.7312643Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:23:41.7313033Z ok (4.912s) 2023-01-11T22:23:41.7313298Z test_single_joinable_disable (__main__.TestJoin) 2023-01-11T22:23:41.7313761Z Tests ``enable=False`` for a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65744 2023-01-11T22:23:41.7314269Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65745 2023-01-11T22:23:41.7314859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:23:41.7315306Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:23:41.7315870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:23:41.7316338Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:23:41.7316891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:23:41.7317331Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:23:41.7317892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:23:41.7318341Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:23:41.7318769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:23:41.7319251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:23:41.7319727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:23:41.7320187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:23:41.7320900Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:23:41.7321580Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:23:41.7321972Z ok (4.912s) 2023-01-11T22:23:41.7322239Z test_single_joinable_main_hooks (__main__.TestJoin) 2023-01-11T22:23:41.7322706Z Tests the main hooks of a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65824 2023-01-11T22:23:41.7323211Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65825 2023-01-11T22:23:41.7323793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:23:41.7324236Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:23:41.7324804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:23:41.7325267Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:23:41.7325819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:23:41.7326257Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:23:41.7326873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:23:41.7327328Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:23:41.7327761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:23:41.7328245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:23:41.7328727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:23:41.7329188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:23:41.7329841Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:23:41.7330800Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:23:41.7331207Z ok (4.935s) 2023-01-11T22:23:41.7331479Z test_single_joinable_post_hooks (__main__.TestJoin) 2023-01-11T22:23:41.7332075Z Tests the post-hooks of a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65904 2023-01-11T22:23:41.7332588Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65905 2023-01-11T22:23:41.7333170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:23:41.7333620Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:23:41.7334186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:23:41.7334651Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:23:41.7335201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:23:41.7335643Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:23:41.7336206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:23:41.7336649Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:23:41.7337081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:23:41.7337562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:23:41.7338133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:23:41.7338593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:23:41.7339245Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:23:41.7339930Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:23:41.7340322Z ok (4.912s) 2023-01-11T22:23:41.7340582Z test_single_joinable_throw (__main__.TestJoin) 2023-01-11T22:23:41.7341041Z Tests ``throw_on_early_termination=True`` for a single ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65984 2023-01-11T22:23:41.7341546Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65985 2023-01-11T22:23:41.7342136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:23:41.7342584Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:23:41.7343149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:23:41.7343616Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:23:41.7344237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:23:41.7344685Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:23:41.7345252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:23:41.7345698Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:23:41.7346133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:23:41.7346620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:23:41.7347101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:23:41.7347560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:23:41.7348208Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:23:41.7348889Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:23:41.7349281Z ok (4.912s) 2023-01-11T22:23:41.7349414Z 2023-01-11T22:23:41.7349682Z ---------------------------------------------------------------------- 2023-01-11T22:23:41.7350010Z Ran 9 tests in 45.823s 2023-01-11T22:23:41.7350172Z 2023-01-11T22:23:41.7350269Z OK 2023-01-11T22:23:41.7350384Z 2023-01-11T22:23:41.7350508Z Generating XML reports... 2023-01-11T22:23:41.7351069Z Generated XML report: test-reports/python-unittest/distributed.algorithms.test_join/TEST-TestJoin-20230111222255.xml 2023-01-11T22:23:41.7351402Z 2023-01-11T22:23:41.7351864Z ##[endgroup] 2023-01-11T22:23:41.7352444Z FINISHED PRINTING LOG FILE of distributed/algorithms/test_join (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-test_join_y0rvk38w) 2023-01-11T22:23:41.7352811Z 2023-01-11T22:23:41.7353094Z Running distributed/_composable/test_fully_shard ... [2023-01-11 22:23:41.725259] 2023-01-11T22:23:41.7353788Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_composable/test_fully_shard.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:23:41.725593] 2023-01-11T22:24:36.8924105Z 2023-01-11T22:24:36.8924915Z Expand the folded group to see the log file of distributed/_composable/test_fully_shard 2023-01-11T22:24:36.8926769Z ##[group]PRINTING LOG FILE of distributed/_composable/test_fully_shard (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_fully_shard_2dgq0xup) 2023-01-11T22:24:36.8927160Z 2023-01-11T22:24:36.8927264Z Running tests... 2023-01-11T22:24:36.8927811Z ---------------------------------------------------------------------- 2023-01-11T22:24:36.8928607Z Test results will be stored in test-reports/python-unittest/distributed._composable.test_fully_shard 2023-01-11T22:24:36.8929163Z test_device_id (__main__.TestFSDPInitialization) 2023-01-11T22:24:36.8929594Z Tests passing a ``device_id``. ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:24:36.8930002Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66099 2023-01-11T22:24:36.8930725Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66100 2023-01-11T22:24:36.8932009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:24:36.8932509Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:24:36.8933351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:24:36.8933845Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:24:36.8934665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:24:36.8935343Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:24:36.8936220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:24:36.8936689Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:24:36.8937372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:24:36.8937895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:24:36.8938792Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:24:36.8939541Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:24:36.8940258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:24:36.8940759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:24:36.8941320Z dist init r=1, world=2 2023-01-11T22:24:36.8941564Z dist init r=0, world=2 2023-01-11T22:24:36.8941831Z ok (5.556s) 2023-01-11T22:24:36.8942156Z test_manual_fully_shard (__main__.TestFSDPInitialization) 2023-01-11T22:24:36.8942851Z Tests manually applying ``fully_shard``. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66178 2023-01-11T22:24:36.8943357Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66179 2023-01-11T22:24:36.8944215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:24:36.8944688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:24:36.8945512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:24:36.8945978Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:24:36.8946820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:24:36.8947274Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:24:36.8948080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:24:36.8948551Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:24:36.8949402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:24:36.8949886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:24:36.8950821Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:24:36.8951518Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:24:36.8952263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:24:36.8952737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:24:36.8953295Z dist init r=1, world=2 2023-01-11T22:24:36.8953582Z dist init r=0, world=2 2023-01-11T22:24:36.8953916Z ok (3.911s) 2023-01-11T22:24:36.8954263Z test_materialize_meta_module (__main__.TestFSDPInitialization) 2023-01-11T22:24:36.8955108Z Tests materializing a meta-device module. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66257 2023-01-11T22:24:36.8955695Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66258 2023-01-11T22:24:36.8956459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:24:36.8957049Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:24:36.8957788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:24:36.8958301Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:24:36.8959083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:24:36.8959528Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:24:36.8960338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:24:36.8960804Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:24:36.8961493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:24:36.8961979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:24:36.8962879Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:24:36.8963622Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:24:36.8964336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:24:36.8964792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:24:36.8965394Z dist init r=1, world=2 2023-01-11T22:24:36.8965650Z dist init r=0, world=2 2023-01-11T22:24:36.8965876Z ok (3.911s) 2023-01-11T22:24:36.8966265Z test_nested_fully_shard_shared_state (__main__.TestFSDPInitialization) 2023-01-11T22:24:36.8966980Z Tests that nested applications of ``fully_shard`` share the expected ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66336 2023-01-11T22:24:36.8967546Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66337 2023-01-11T22:24:36.8968358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:24:36.8968814Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:24:36.8969648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:24:36.8970121Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:24:36.8971268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:24:36.8971893Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:24:36.8972543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:24:36.8973125Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:24:36.8973702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:24:36.8974197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:24:36.8975096Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:24:36.8976009Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:24:36.8976553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:24:36.8977191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:24:36.8977665Z dist init r=1, world=2 2023-01-11T22:24:36.8977920Z dist init r=0, world=2 2023-01-11T22:24:36.8978161Z ok (4.411s) 2023-01-11T22:24:36.8978746Z test_policy (__main__.TestFSDPInitialization) 2023-01-11T22:24:36.8979414Z Tests passing a ``policy`` for pseudo-auto-wrapping. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66415 2023-01-11T22:24:36.8980196Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66416 2023-01-11T22:24:36.8980823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:24:36.8981478Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:24:36.8982080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:24:36.8982768Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:24:36.8983380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:24:36.8983938Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:24:36.8984641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:24:36.8985197Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:24:36.8985783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:24:36.8986284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:24:36.8987187Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:24:36.8988068Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:24:36.8988629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:24:36.8989107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:24:36.8989594Z dist init r=1, world=2 2023-01-11T22:24:36.8989944Z dist init r=0, world=2 2023-01-11T22:24:36.8990193Z ok (4.011s) 2023-01-11T22:24:36.8990506Z test_sync_module_states (__main__.TestFSDPInitialization) 2023-01-11T22:24:36.8990985Z Tests passing ``sync_module_states=True``. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66494 2023-01-11T22:24:36.8991466Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66495 2023-01-11T22:24:36.8992453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:24:36.8992910Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:24:36.8993468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:24:36.8993943Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:24:36.8994527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:24:36.8994968Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:24:36.8995515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:24:36.8996115Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:24:36.8996696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:24:36.8997193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:24:36.8997832Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:24:36.8998586Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:24:36.8999117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:24:36.8999571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:24:36.8999930Z dist init r=1, world=2 2023-01-11T22:24:36.9000178Z dist init r=0, world=2 2023-01-11T22:24:36.9000414Z ok (4.010s) 2023-01-11T22:24:36.9000727Z test_state_dict_save_load_flow (__main__.TestFSDPModelCheckpointing) 2023-01-11T22:24:36.9001300Z E2E test of save + load with rank0_only + CPU offload for TransformerWithSharedParams ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66573 2023-01-11T22:24:36.9001865Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66574 2023-01-11T22:24:36.9002453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:24:36.9002905Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:24:36.9003757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:24:36.9004232Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:24:36.9004791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:24:36.9005234Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:24:36.9005805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:24:36.9006247Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:24:36.9006701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:24:36.9007197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:24:36.9007857Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:24:36.9008516Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:24:36.9009035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:24:36.9009505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:24:36.9009938Z dist init r=0, world=2 2023-01-11T22:24:36.9010172Z dist init r=1, world=2 2023-01-11T22:24:36.9010675Z ok (5.313s) 2023-01-11T22:24:36.9011032Z test_state_dict_save_load_root_fully_shard (__main__.TestFSDPModelCheckpointing) 2023-01-11T22:24:36.9011562Z Tests that the full state dict saved from a module with ``fully_shard`` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66652 2023-01-11T22:24:36.9012090Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66653 2023-01-11T22:24:36.9012703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:24:36.9013132Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:24:36.9013703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:24:36.9014174Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:24:36.9014747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:24:36.9015171Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:24:36.9015747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:24:36.9016288Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:24:36.9016749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:24:36.9017500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:24:36.9018168Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:24:36.9018853Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:24:36.9019356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:24:36.9019826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:24:36.9020175Z dist init r=1, world=2 2023-01-11T22:24:36.9020425Z dist init r=0, world=2 2023-01-11T22:24:36.9020646Z ok (4.010s) 2023-01-11T22:24:36.9021004Z test_state_dict_save_load_submodule_fully_shard (__main__.TestFSDPModelCheckpointing) 2023-01-11T22:24:36.9021558Z Tests that the full state dict saved from a module with ``fully_shard`` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66731 2023-01-11T22:24:36.9022069Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66732 2023-01-11T22:24:36.9022677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:24:36.9023128Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:24:36.9023699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:24:36.9024146Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:24:36.9024717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:24:36.9025164Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:24:36.9025714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:24:36.9026177Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:24:36.9026626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:24:36.9027120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:24:36.9027859Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:24:36.9028543Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:24:36.9029064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:24:36.9029539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:24:36.9029877Z dist init r=1, world=2 2023-01-11T22:24:36.9030130Z dist init r=0, world=2 2023-01-11T22:24:36.9030370Z ok (4.111s) 2023-01-11T22:24:36.9030742Z test_optim_state_dict_save_load (__main__.TestFSDPOptimStateDict) ... skip: The test currently fails on CI. (0.001s) 2023-01-11T22:24:36.9031285Z test_optim_state_dict_submodule_fully_shard (__main__.TestFSDPOptimStateDict) ... skip: The test currently fails on CI. (0.001s) 2023-01-11T22:24:36.9031720Z test_training (__main__.TestFSDPRuntime) 2023-01-11T22:24:36.9032177Z Tests training (forward, backward, optimizer). ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66810 2023-01-11T22:24:36.9032670Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66811 2023-01-11T22:24:36.9033341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:24:36.9033798Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:24:36.9034351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:24:36.9034820Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:24:36.9035395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:24:36.9035838Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:24:36.9036385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:24:36.9036865Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:24:36.9037294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:24:36.9037770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:24:36.9038420Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:24:36.9039101Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:24:36.9039617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:24:36.9040067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:24:36.9040412Z dist init r=0, world=2 2023-01-11T22:24:36.9040659Z dist init r=1, world=2 2023-01-11T22:24:36.9040881Z ok (4.611s) 2023-01-11T22:24:36.9041180Z test_unshard_reshard_order (__main__.TestFSDPRuntime) 2023-01-11T22:24:36.9041686Z Tests that the unshard/reshard order matches between ``fully_shard`` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66893 2023-01-11T22:24:36.9042213Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66894 2023-01-11T22:24:36.9042801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:24:36.9043248Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:24:36.9044115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:24:36.9044651Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:24:36.9045523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:24:36.9045967Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:24:36.9046534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:24:36.9046982Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:24:36.9047435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:24:36.9047930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:24:36.9048580Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:24:36.9049251Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:24:36.9049771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:24:36.9050237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:24:36.9050911Z dist init r=0, world=2 2023-01-11T22:24:36.9051171Z dist init r=1, world=2 2023-01-11T22:24:36.9051530Z ok (4.511s) 2023-01-11T22:24:36.9051981Z test_float16_on_one_submodule (__main__.TestMixedPrecision) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66976 2023-01-11T22:24:36.9052491Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66977 2023-01-11T22:24:36.9053106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:24:36.9053555Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:24:36.9054114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:24:36.9054582Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:24:36.9055153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:24:36.9055596Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:24:36.9056148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:24:36.9056609Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:24:36.9057052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:24:36.9057525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:24:36.9058181Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:24:36.9058860Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:24:36.9059378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:24:36.9059833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:24:36.9060181Z dist init r=0, world=2 2023-01-11T22:24:36.9060431Z dist init r=1, world=2 2023-01-11T22:24:36.9060649Z ok (4.412s) 2023-01-11T22:24:36.9060799Z 2023-01-11T22:24:36.9061067Z ---------------------------------------------------------------------- 2023-01-11T22:24:36.9061399Z Ran 14 tests in 52.783s 2023-01-11T22:24:36.9061567Z 2023-01-11T22:24:36.9061676Z OK (skipped=2) 2023-01-11T22:24:36.9061814Z 2023-01-11T22:24:36.9061939Z Generating XML reports... 2023-01-11T22:24:36.9062648Z Generated XML report: test-reports/python-unittest/distributed._composable.test_fully_shard/TEST-TestFSDPInitialization-20230111222343.xml 2023-01-11T22:24:36.9063480Z Generated XML report: test-reports/python-unittest/distributed._composable.test_fully_shard/TEST-TestFSDPModelCheckpointing-20230111222343.xml 2023-01-11T22:24:36.9064273Z Generated XML report: test-reports/python-unittest/distributed._composable.test_fully_shard/TEST-TestFSDPRuntime-20230111222343.xml 2023-01-11T22:24:36.9065022Z Generated XML report: test-reports/python-unittest/distributed._composable.test_fully_shard/TEST-TestMixedPrecision-20230111222343.xml 2023-01-11T22:24:36.9065805Z Generated XML report: test-reports/python-unittest/distributed._composable.test_fully_shard/TEST-TestFSDPOptimStateDict-20230111222343.xml 2023-01-11T22:24:36.9066170Z 2023-01-11T22:24:36.9066607Z ##[endgroup] 2023-01-11T22:24:36.9067213Z FINISHED PRINTING LOG FILE of distributed/_composable/test_fully_shard (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_fully_shard_2dgq0xup) 2023-01-11T22:24:36.9067589Z 2023-01-11T22:24:36.9067848Z Running distributed/test_pg_wrapper ... [2023-01-11 22:24:36.892595] 2023-01-11T22:24:36.9068535Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_pg_wrapper.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:24:36.892862] 2023-01-11T22:26:19.1270305Z 2023-01-11T22:26:19.1271497Z Expand the folded group to see the log file of distributed/test_pg_wrapper 2023-01-11T22:26:19.1272942Z ##[group]PRINTING LOG FILE of distributed/test_pg_wrapper (/var/lib/jenkins/workspace/test/test-reports/distributed-test_pg_wrapper_6_ex2316) 2023-01-11T22:26:19.1273782Z 2023-01-11T22:26:19.1274101Z 2023-01-11T22:26:19.1275570Z , <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch_cuda>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch_cuda_debug_mode>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch_debug_mode>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch_cuda>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch_cuda_debug_mode>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch_debug_mode>]> 2023-01-11T22:26:19.1277220Z test_collective_hang (__main__.ProcessGroupGlooWrapperTest) 2023-01-11T22:26:19.1277665Z test_collective_shape_mismatch (__main__.ProcessGroupGlooWrapperTest) 2023-01-11T22:26:19.1278102Z test_collective_shape_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest) 2023-01-11T22:26:19.1278568Z test_collective_shape_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest) 2023-01-11T22:26:19.1279071Z test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest) 2023-01-11T22:26:19.1279657Z test_collectives_op_mismatch (__main__.ProcessGroupGlooWrapperTest) 2023-01-11T22:26:19.1280070Z test_collectives_op_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest) 2023-01-11T22:26:19.1280531Z test_collectives_op_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest) 2023-01-11T22:26:19.1281011Z test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest) 2023-01-11T22:26:19.1282025Z , <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collective_shape_mismatch>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collective_shape_mismatch_debug_mode>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collectives_op_mismatch>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collectives_op_mismatch_debug_mode>]> 2023-01-11T22:26:19.1283195Z test_collective_hang (__main__.ProcessGroupNCCLWrapperTest) 2023-01-11T22:26:19.1283611Z test_collective_shape_mismatch (__main__.ProcessGroupNCCLWrapperTest) 2023-01-11T22:26:19.1284068Z test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest) 2023-01-11T22:26:19.1284520Z test_collectives_op_mismatch (__main__.ProcessGroupNCCLWrapperTest) 2023-01-11T22:26:19.1284967Z test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest) 2023-01-11T22:26:19.1286033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1286654Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1287985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1288742Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1288983Z 2023-01-11T22:26:19.1295271Z Running tests... 2023-01-11T22:26:19.1296164Z ---------------------------------------------------------------------- 2023-01-11T22:26:19.1297207Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2023-01-11T22:26:19.1298162Z test_collective_hang (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:26:19.1299219Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67127 2023-01-11T22:26:19.1300023Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67128 2023-01-11T22:26:19.1300837Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 67129 2023-01-11T22:26:19.1301632Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 67130 2023-01-11T22:26:19.1302715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1303511Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1304536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1305330Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1306489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1307210Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1307769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1308218Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1308769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1309236Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1309816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1310266Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1310837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1311279Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1311850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1312318Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1312735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T22:26:19.1313208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T22:26:19.1313773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:26:19.1314236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:26:19.1314714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:26:19.1315202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:26:19.1315671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2023-01-11T22:26:19.1316153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2023-01-11T22:26:19.1316808Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2023-01-11T22:26:19.1317493Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2023-01-11T22:26:19.1318159Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2023-01-11T22:26:19.1318842Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2023-01-11T22:26:19.1319360Z [E ProcessGroupGloo.cpp:2803] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms 2023-01-11T22:26:19.1319877Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 2000 ms 2023-01-11T22:26:19.1320447Z [E ProcessGroupGloo.cpp:138] Rank 2 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank. 2023-01-11T22:26:19.1321131Z [E ProcessGroupGloo.cpp:138] Rank 3 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank. 2023-01-11T22:26:19.1321586Z ok (4.129s) 2023-01-11T22:26:19.1321733Z 2023-01-11T22:26:19.1322004Z ---------------------------------------------------------------------- 2023-01-11T22:26:19.1322315Z Ran 1 test in 4.129s 2023-01-11T22:26:19.1322477Z 2023-01-11T22:26:19.1322819Z OK 2023-01-11T22:26:19.1322956Z 2023-01-11T22:26:19.1323080Z Generating XML reports... 2023-01-11T22:26:19.1323688Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111222440.xml 2023-01-11T22:26:19.1324420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1324867Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1325437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1325894Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1326127Z 2023-01-11T22:26:19.1326234Z Running tests... 2023-01-11T22:26:19.1326633Z ---------------------------------------------------------------------- 2023-01-11T22:26:19.1327136Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2023-01-11T22:26:19.1327660Z test_collective_shape_mismatch (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:26:19.1328168Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67334 2023-01-11T22:26:19.1328618Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67335 2023-01-11T22:26:19.1329066Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 67336 2023-01-11T22:26:19.1329473Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 67337 2023-01-11T22:26:19.1330080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1331296Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1331886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1332355Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1332923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1333353Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1333921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1334377Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1334947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1335370Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1335939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1336397Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1336969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1337382Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1338087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1338561Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1338974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:26:19.1339442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:26:19.1339912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T22:26:19.1340360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T22:26:19.1340843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:26:19.1341331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:26:19.1341814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2023-01-11T22:26:19.1342276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2023-01-11T22:26:19.1342931Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2023-01-11T22:26:19.1343614Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2023-01-11T22:26:19.1344303Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2023-01-11T22:26:19.1344951Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2023-01-11T22:26:19.1345345Z ok (4.241s) 2023-01-11T22:26:19.1345493Z 2023-01-11T22:26:19.1345761Z ---------------------------------------------------------------------- 2023-01-11T22:26:19.1346073Z Ran 1 test in 4.241s 2023-01-11T22:26:19.1346231Z 2023-01-11T22:26:19.1346322Z OK 2023-01-11T22:26:19.1346455Z 2023-01-11T22:26:19.1346577Z Generating XML reports... 2023-01-11T22:26:19.1347201Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111222447.xml 2023-01-11T22:26:19.1347909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1348434Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1349006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1349455Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1349680Z 2023-01-11T22:26:19.1349787Z Running tests... 2023-01-11T22:26:19.1350187Z ---------------------------------------------------------------------- 2023-01-11T22:26:19.1350711Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2023-01-11T22:26:19.1351230Z test_collective_shape_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:26:19.1351736Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67541 2023-01-11T22:26:19.1352179Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67542 2023-01-11T22:26:19.1352619Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 67543 2023-01-11T22:26:19.1353044Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 67544 2023-01-11T22:26:19.1353643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1354089Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1354677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1355151Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1355720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1356161Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1356708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1357172Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1357736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1358154Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1358716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1359182Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1359750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1360167Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1360729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1361192Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1361603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T22:26:19.1362072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T22:26:19.1362528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:26:19.1362995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:26:19.1363363Z skip: Need at least 4 CUDA devices (3.963s) 2023-01-11T22:26:19.1363558Z 2023-01-11T22:26:19.1363826Z ---------------------------------------------------------------------- 2023-01-11T22:26:19.1364148Z Ran 1 test in 3.964s 2023-01-11T22:26:19.1364311Z 2023-01-11T22:26:19.1364417Z OK (skipped=1) 2023-01-11T22:26:19.1364554Z 2023-01-11T22:26:19.1364676Z Generating XML reports... 2023-01-11T22:26:19.1365289Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111222453.xml 2023-01-11T22:26:19.1366068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1366494Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1367058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1367525Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1367754Z 2023-01-11T22:26:19.1367862Z Running tests... 2023-01-11T22:26:19.1368248Z ---------------------------------------------------------------------- 2023-01-11T22:26:19.1368772Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2023-01-11T22:26:19.1369319Z test_collective_shape_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:26:19.1369831Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67712 2023-01-11T22:26:19.1370274Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67713 2023-01-11T22:26:19.1371216Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 67714 2023-01-11T22:26:19.1371658Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 67715 2023-01-11T22:26:19.1372332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1372788Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1373397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1373847Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1374421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1374871Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1375435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1375877Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1376452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1376895Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1377462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1377905Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1378475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1378916Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1379459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1379919Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1380348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T22:26:19.1380823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:26:19.1381269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T22:26:19.1381728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:26:19.1382117Z skip: Need at least 4 CUDA devices (4.041s) 2023-01-11T22:26:19.1382311Z 2023-01-11T22:26:19.1382562Z ---------------------------------------------------------------------- 2023-01-11T22:26:19.1382956Z Ran 1 test in 4.041s 2023-01-11T22:26:19.1383117Z 2023-01-11T22:26:19.1383225Z OK (skipped=1) 2023-01-11T22:26:19.1383381Z 2023-01-11T22:26:19.1383507Z Generating XML reports... 2023-01-11T22:26:19.1384108Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111222459.xml 2023-01-11T22:26:19.1384840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1385283Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1385831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1386294Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1386526Z 2023-01-11T22:26:19.1386632Z Running tests... 2023-01-11T22:26:19.1387032Z ---------------------------------------------------------------------- 2023-01-11T22:26:19.1387541Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2023-01-11T22:26:19.1388083Z test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:26:19.1388602Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67883 2023-01-11T22:26:19.1389028Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67884 2023-01-11T22:26:19.1389512Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 67885 2023-01-11T22:26:19.1389950Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 67886 2023-01-11T22:26:19.1390552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1390981Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1391549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1392019Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1392588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1393009Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1393578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1394035Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1394588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1395027Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1395589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1396051Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1396600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1397038Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1397606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1398044Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1398470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:26:19.1398936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T22:26:19.1399396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T22:26:19.1399895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:26:19.1400371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:26:19.1400858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:26:19.1401340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2023-01-11T22:26:19.1401810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2023-01-11T22:26:19.1402458Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2023-01-11T22:26:19.1403146Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2023-01-11T22:26:19.1403807Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2023-01-11T22:26:19.1404489Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2023-01-11T22:26:19.1405014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T22:26:19.1405496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2023-01-11T22:26:19.1406025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T22:26:19.1406513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3 2023-01-11T22:26:19.1407153Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2023-01-11T22:26:19.1407822Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2023-01-11T22:26:19.1408480Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2023-01-11T22:26:19.1409153Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2023-01-11T22:26:19.1409548Z ok (4.229s) 2023-01-11T22:26:19.1409696Z 2023-01-11T22:26:19.1409957Z ---------------------------------------------------------------------- 2023-01-11T22:26:19.1410264Z Ran 1 test in 4.229s 2023-01-11T22:26:19.1410921Z 2023-01-11T22:26:19.1411017Z OK 2023-01-11T22:26:19.1411152Z 2023-01-11T22:26:19.1411278Z Generating XML reports... 2023-01-11T22:26:19.1411891Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111222506.xml 2023-01-11T22:26:19.1412614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1413062Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1413638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1414088Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1414317Z 2023-01-11T22:26:19.1414426Z Running tests... 2023-01-11T22:26:19.1414822Z ---------------------------------------------------------------------- 2023-01-11T22:26:19.1415329Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2023-01-11T22:26:19.1415853Z test_collectives_op_mismatch (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:26:19.1416357Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68102 2023-01-11T22:26:19.1416804Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68103 2023-01-11T22:26:19.1417225Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 68104 2023-01-11T22:26:19.1417747Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 68105 2023-01-11T22:26:19.1418351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1418783Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1419353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1419819Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1420388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1420810Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1421367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1421828Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1422389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1422807Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1423371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1423893Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1424454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1424898Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1425460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1425922Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1426337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:26:19.1426804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:26:19.1427266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T22:26:19.1427710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T22:26:19.1428194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:26:19.1428686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:26:19.1429169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2023-01-11T22:26:19.1429631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2023-01-11T22:26:19.1430282Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2023-01-11T22:26:19.1430966Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2023-01-11T22:26:19.1431641Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2023-01-11T22:26:19.1432303Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2023-01-11T22:26:19.1432696Z ok (4.253s) 2023-01-11T22:26:19.1432844Z 2023-01-11T22:26:19.1433106Z ---------------------------------------------------------------------- 2023-01-11T22:26:19.1433409Z Ran 1 test in 4.253s 2023-01-11T22:26:19.1433571Z 2023-01-11T22:26:19.1433661Z OK 2023-01-11T22:26:19.1433796Z 2023-01-11T22:26:19.1433918Z Generating XML reports... 2023-01-11T22:26:19.1434593Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111222512.xml 2023-01-11T22:26:19.1435301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1435745Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1436309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1436758Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1436984Z 2023-01-11T22:26:19.1437093Z Running tests... 2023-01-11T22:26:19.1437491Z ---------------------------------------------------------------------- 2023-01-11T22:26:19.1438012Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2023-01-11T22:26:19.1438524Z test_collectives_op_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:26:19.1439035Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68309 2023-01-11T22:26:19.1439481Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68310 2023-01-11T22:26:19.1439924Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 68311 2023-01-11T22:26:19.1440348Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 68312 2023-01-11T22:26:19.1441000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1441451Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1442001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1442468Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1443039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1443484Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1444029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1444492Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1445067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1445488Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1446046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1446507Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1447077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1447502Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1448062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1448521Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1448929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T22:26:19.1449402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:26:19.1449859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:26:19.1450744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T22:26:19.1451176Z skip: Need at least 4 CUDA devices (4.014s) 2023-01-11T22:26:19.1451370Z 2023-01-11T22:26:19.1451638Z ---------------------------------------------------------------------- 2023-01-11T22:26:19.1452052Z Ran 1 test in 4.014s 2023-01-11T22:26:19.1452214Z 2023-01-11T22:26:19.1452322Z OK (skipped=1) 2023-01-11T22:26:19.1452457Z 2023-01-11T22:26:19.1452579Z Generating XML reports... 2023-01-11T22:26:19.1453201Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111222519.xml 2023-01-11T22:26:19.1453927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1454357Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1454920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1455386Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1455615Z 2023-01-11T22:26:19.1455723Z Running tests... 2023-01-11T22:26:19.1456102Z ---------------------------------------------------------------------- 2023-01-11T22:26:19.1456630Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2023-01-11T22:26:19.1457173Z test_collectives_op_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:26:19.1457678Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68480 2023-01-11T22:26:19.1458188Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68481 2023-01-11T22:26:19.1458633Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 68482 2023-01-11T22:26:19.1459069Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 68483 2023-01-11T22:26:19.1459654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1460099Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1460666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1461117Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1461688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1462122Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1462689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1463134Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1463707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1464148Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1464713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1465158Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1465730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1466165Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1466709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1467173Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1467601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:26:19.1468069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T22:26:19.1468513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:26:19.1469022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T22:26:19.1469414Z skip: Need at least 4 CUDA devices (3.963s) 2023-01-11T22:26:19.1469603Z 2023-01-11T22:26:19.1469853Z ---------------------------------------------------------------------- 2023-01-11T22:26:19.1470180Z Ran 1 test in 3.964s 2023-01-11T22:26:19.1470342Z 2023-01-11T22:26:19.1470449Z OK (skipped=1) 2023-01-11T22:26:19.1470603Z 2023-01-11T22:26:19.1470728Z Generating XML reports... 2023-01-11T22:26:19.1471329Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111222525.xml 2023-01-11T22:26:19.1472052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1472492Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1473044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1473557Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1473786Z 2023-01-11T22:26:19.1473894Z Running tests... 2023-01-11T22:26:19.1474293Z ---------------------------------------------------------------------- 2023-01-11T22:26:19.1474801Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2023-01-11T22:26:19.1475391Z test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:26:19.1475913Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68651 2023-01-11T22:26:19.1476342Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68652 2023-01-11T22:26:19.1476786Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 68653 2023-01-11T22:26:19.1477218Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 68654 2023-01-11T22:26:19.1477823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1478250Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1478806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1479244Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1479811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1480257Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1480834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1481300Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1481854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1482297Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1482859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1483318Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1483869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1484311Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1484873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1485315Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1485745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T22:26:19.1486266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:26:19.1486729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:26:19.1487174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T22:26:19.1487649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:26:19.1488142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2023-01-11T22:26:19.1488623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:26:19.1489087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2023-01-11T22:26:19.1489734Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2023-01-11T22:26:19.1490908Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2023-01-11T22:26:19.1491590Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2023-01-11T22:26:19.1492265Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2023-01-11T22:26:19.1492874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T22:26:19.1493367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T22:26:19.1493831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3 2023-01-11T22:26:19.1494308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2023-01-11T22:26:19.1494954Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2023-01-11T22:26:19.1495628Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2023-01-11T22:26:19.1496285Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2023-01-11T22:26:19.1496962Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2023-01-11T22:26:19.1497357Z ok (4.351s) 2023-01-11T22:26:19.1497505Z 2023-01-11T22:26:19.1497773Z ---------------------------------------------------------------------- 2023-01-11T22:26:19.1498081Z Ran 1 test in 4.351s 2023-01-11T22:26:19.1498242Z 2023-01-11T22:26:19.1498330Z OK 2023-01-11T22:26:19.1498465Z 2023-01-11T22:26:19.1498588Z Generating XML reports... 2023-01-11T22:26:19.1499188Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111222531.xml 2023-01-11T22:26:19.1499918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1500364Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1500936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1501389Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1501618Z 2023-01-11T22:26:19.1501728Z Running tests... 2023-01-11T22:26:19.1502126Z ---------------------------------------------------------------------- 2023-01-11T22:26:19.1502631Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2023-01-11T22:26:19.1503142Z test_collective_hang (__main__.ProcessGroupNCCLWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:26:19.1503710Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68870 2023-01-11T22:26:19.1504157Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68871 2023-01-11T22:26:19.1504742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1505186Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1505756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1506206Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1506779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1507214Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1507779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1508229Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1508664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:26:19.1509150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:26:19.1509684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:26:19.1510151Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:26:19.1510803Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:26:19.1511487Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:26:19.1511985Z [E ProcessGroupGloo.cpp:2803] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms 2023-01-11T22:26:19.1512454Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 2000 ms 2023-01-11T22:26:19.1512788Z ok (3.934s) 2023-01-11T22:26:19.1512935Z 2023-01-11T22:26:19.1513197Z ---------------------------------------------------------------------- 2023-01-11T22:26:19.1513504Z Ran 1 test in 3.934s 2023-01-11T22:26:19.1513664Z 2023-01-11T22:26:19.1513756Z OK 2023-01-11T22:26:19.1513893Z 2023-01-11T22:26:19.1514019Z Generating XML reports... 2023-01-11T22:26:19.1514627Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20230111222538.xml 2023-01-11T22:26:19.1515353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1515801Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1516364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1516814Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1517044Z 2023-01-11T22:26:19.1517152Z Running tests... 2023-01-11T22:26:19.1517548Z ---------------------------------------------------------------------- 2023-01-11T22:26:19.1518054Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2023-01-11T22:26:19.1518584Z test_collective_shape_mismatch (__main__.ProcessGroupNCCLWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:26:19.1519090Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68983 2023-01-11T22:26:19.1519532Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68984 2023-01-11T22:26:19.1520115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1520559Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1521178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1521644Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1522198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1522642Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1523205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1523650Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1524084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:26:19.1524569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:26:19.1525055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:26:19.1525513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:26:19.1526164Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:26:19.1526907Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:26:19.1527289Z ok (6.767s) 2023-01-11T22:26:19.1527435Z 2023-01-11T22:26:19.1527699Z ---------------------------------------------------------------------- 2023-01-11T22:26:19.1528024Z Ran 1 test in 6.767s 2023-01-11T22:26:19.1528185Z 2023-01-11T22:26:19.1528278Z OK 2023-01-11T22:26:19.1528392Z 2023-01-11T22:26:19.1528513Z Generating XML reports... 2023-01-11T22:26:19.1529126Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20230111222544.xml 2023-01-11T22:26:19.1529854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1530684Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1531341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1531813Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1532046Z 2023-01-11T22:26:19.1532155Z Running tests... 2023-01-11T22:26:19.1532535Z ---------------------------------------------------------------------- 2023-01-11T22:26:19.1533056Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2023-01-11T22:26:19.1533600Z test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:26:19.1534128Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69113 2023-01-11T22:26:19.1534556Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69114 2023-01-11T22:26:19.1535152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1535595Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1536149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1536613Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1537184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1537627Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1538174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1538731Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1539162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:26:19.1539613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:26:19.1540094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:26:19.1540585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:26:19.1541236Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:26:19.1541900Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:26:19.1542426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T22:26:19.1542914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T22:26:19.1543553Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:26:19.1544214Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:26:19.1544670Z ok (6.853s) 2023-01-11T22:26:19.1544825Z 2023-01-11T22:26:19.1545088Z ---------------------------------------------------------------------- 2023-01-11T22:26:19.1545397Z Ran 1 test in 6.853s 2023-01-11T22:26:19.1545557Z 2023-01-11T22:26:19.1545650Z OK 2023-01-11T22:26:19.1545782Z 2023-01-11T22:26:19.1545902Z Generating XML reports... 2023-01-11T22:26:19.1546518Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20230111222553.xml 2023-01-11T22:26:19.1547228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1547675Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1548242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1548689Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1548913Z 2023-01-11T22:26:19.1549024Z Running tests... 2023-01-11T22:26:19.1549423Z ---------------------------------------------------------------------- 2023-01-11T22:26:19.1549943Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2023-01-11T22:26:19.1550447Z test_collectives_op_mismatch (__main__.ProcessGroupNCCLWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:26:19.1550952Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69253 2023-01-11T22:26:19.1551404Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69254 2023-01-11T22:26:19.1552007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1552436Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1553001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1553471Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1554026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1554470Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1555032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1555551Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1555970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:26:19.1556455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:26:19.1556936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:26:19.1557402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:26:19.1558047Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:26:19.1558725Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:26:19.1559120Z ok (6.642s) 2023-01-11T22:26:19.1559267Z 2023-01-11T22:26:19.1559512Z ---------------------------------------------------------------------- 2023-01-11T22:26:19.1559840Z Ran 1 test in 6.642s 2023-01-11T22:26:19.1560000Z 2023-01-11T22:26:19.1560093Z OK 2023-01-11T22:26:19.1560226Z 2023-01-11T22:26:19.1560329Z Generating XML reports... 2023-01-11T22:26:19.1560951Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20230111222602.xml 2023-01-11T22:26:19.1561731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1562183Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1562733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1563202Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1563433Z 2023-01-11T22:26:19.1563541Z Running tests... 2023-01-11T22:26:19.1563937Z ---------------------------------------------------------------------- 2023-01-11T22:26:19.1564447Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2023-01-11T22:26:19.1564984Z test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:26:19.1565498Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69383 2023-01-11T22:26:19.1565926Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69384 2023-01-11T22:26:19.1566522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1566970Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1567537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1567984Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1568560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:26:19.1569004Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:26:19.1569550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:26:19.1570011Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:26:19.1570912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:26:19.1571396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:26:19.1587152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:26:19.1588096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:26:19.1589299Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:26:19.1590700Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:26:19.1591459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2023-01-11T22:26:19.1591933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2023-01-11T22:26:19.1592600Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:26:19.1593279Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2023-01-11T22:26:19.1593671Z ok (6.824s) 2023-01-11T22:26:19.1593818Z 2023-01-11T22:26:19.1594068Z ---------------------------------------------------------------------- 2023-01-11T22:26:19.1594390Z Ran 1 test in 6.825s 2023-01-11T22:26:19.1594615Z 2023-01-11T22:26:19.1594706Z OK 2023-01-11T22:26:19.1594837Z 2023-01-11T22:26:19.1594943Z Generating XML reports... 2023-01-11T22:26:19.1595568Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20230111222611.xml 2023-01-11T22:26:19.1595945Z 2023-01-11T22:26:19.1596393Z ##[endgroup] 2023-01-11T22:26:19.1597065Z FINISHED PRINTING LOG FILE of distributed/test_pg_wrapper (/var/lib/jenkins/workspace/test/test-reports/distributed-test_pg_wrapper_6_ex2316) 2023-01-11T22:26:19.1597417Z 2023-01-11T22:26:19.1597681Z Running distributed/fsdp/test_fsdp_checkpoint ... [2023-01-11 22:26:19.128221] 2023-01-11T22:26:19.1598369Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:26:19.128502] 2023-01-11T22:27:39.9504541Z 2023-01-11T22:27:39.9505347Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_checkpoint 2023-01-11T22:27:39.9506324Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_checkpoint_o661x9eo) 2023-01-11T22:27:39.9509400Z 2023-01-11T22:27:39.9509999Z Running tests... 2023-01-11T22:27:39.9510883Z ---------------------------------------------------------------------- 2023-01-11T22:27:39.9511483Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint 2023-01-11T22:27:39.9512185Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:27:39.9512861Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69523 2023-01-11T22:27:39.9513578Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69524 2023-01-11T22:27:39.9514213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:27:39.9514702Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:27:39.9515266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:27:39.9515748Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:27:39.9516337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:27:39.9516796Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:27:39.9517355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:27:39.9517822Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:27:39.9518273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:27:39.9519004Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:27:39.9519674Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:27:39.9520364Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:27:39.9520889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:27:39.9521345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:27:39.9521703Z dist init r=0, world=2 2023-01-11T22:27:39.9521959Z dist init r=1, world=2 2023-01-11T22:27:39.9522209Z ok (6.162s) 2023-01-11T22:27:39.9523407Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69606 2023-01-11T22:27:39.9524183Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69607 2023-01-11T22:27:39.9524841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:27:39.9525297Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:27:39.9526110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:27:39.9526613Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:27:39.9527184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:27:39.9527633Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:27:39.9528210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:27:39.9528696Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:27:39.9529135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:27:39.9529640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:27:39.9530596Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:27:39.9531307Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:27:39.9531828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:27:39.9532296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:27:39.9532652Z dist init r=0, world=2 2023-01-11T22:27:39.9532893Z dist init r=1, world=2 2023-01-11T22:27:39.9533132Z ok (4.513s) 2023-01-11T22:27:39.9533689Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69689 2023-01-11T22:27:39.9534324Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69690 2023-01-11T22:27:39.9534942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:27:39.9535396Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:27:39.9535969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:27:39.9536424Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:27:39.9536997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:27:39.9537561Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:27:39.9538134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:27:39.9538583Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:27:39.9539037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:27:39.9539532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:27:39.9540161Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:27:39.9540851Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:27:39.9541368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:27:39.9541844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:27:39.9542185Z dist init r=1, world=2 2023-01-11T22:27:39.9542437Z dist init r=0, world=2 2023-01-11T22:27:39.9542677Z ok (4.613s) 2023-01-11T22:27:39.9543297Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69772 2023-01-11T22:27:39.9543960Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69773 2023-01-11T22:27:39.9544572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:27:39.9545020Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:27:39.9545571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:27:39.9546042Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:27:39.9546618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:27:39.9547058Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:27:39.9547613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:27:39.9548073Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:27:39.9548521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:27:39.9548993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:27:39.9549649Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:27:39.9550470Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:27:39.9550995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:27:39.9551443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:27:39.9551791Z dist init r=1, world=2 2023-01-11T22:27:39.9552045Z dist init r=0, world=2 2023-01-11T22:27:39.9552269Z ok (4.613s) 2023-01-11T22:27:39.9552833Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69855 2023-01-11T22:27:39.9553480Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69856 2023-01-11T22:27:39.9554091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:27:39.9554593Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:27:39.9555173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:27:39.9555642Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:27:39.9556215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:27:39.9556640Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:27:39.9557206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:27:39.9557667Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:27:39.9558097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:27:39.9558593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:27:39.9559244Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:27:39.9559932Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:27:39.9560489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:27:39.9560969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:27:39.9561327Z dist init r=0, world=2 2023-01-11T22:27:39.9561579Z dist init r=1, world=2 2023-01-11T22:27:39.9561803Z ok (4.513s) 2023-01-11T22:27:39.9562358Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69938 2023-01-11T22:27:39.9563013Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69939 2023-01-11T22:27:39.9563612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:27:39.9564068Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:27:39.9564645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:27:39.9565117Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:27:39.9565675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:27:39.9566120Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:27:39.9566689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:27:39.9567134Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:27:39.9567581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:27:39.9568075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:27:39.9568733Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:27:39.9569401Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:27:39.9569919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:27:39.9570554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:27:39.9570921Z dist init r=1, world=2 2023-01-11T22:27:39.9571250Z dist init r=0, world=2 2023-01-11T22:27:39.9571490Z ok (4.613s) 2023-01-11T22:27:39.9572044Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70021 2023-01-11T22:27:39.9572683Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70022 2023-01-11T22:27:39.9573303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:27:39.9573758Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:27:39.9574333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:27:39.9574786Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:27:39.9575362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:27:39.9575808Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:27:39.9576361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:27:39.9576866Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:27:39.9577454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:27:39.9577962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:27:39.9578602Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:27:39.9579292Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:27:39.9579815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:27:39.9580290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:27:39.9580629Z dist init r=0, world=2 2023-01-11T22:27:39.9580881Z dist init r=1, world=2 2023-01-11T22:27:39.9581124Z ok (4.512s) 2023-01-11T22:27:39.9581660Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70104 2023-01-11T22:27:39.9582315Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70105 2023-01-11T22:27:39.9582930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:27:39.9583384Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:27:39.9583933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:27:39.9584406Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:27:39.9584983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:27:39.9585409Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:27:39.9585983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:27:39.9586447Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:27:39.9586899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:27:39.9587373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:27:39.9588029Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:27:39.9588784Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:27:39.9589303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:27:39.9589752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:27:39.9590101Z dist init r=0, world=2 2023-01-11T22:27:39.9590358Z dist init r=1, world=2 2023-01-11T22:27:39.9590580Z ok (4.612s) 2023-01-11T22:27:39.9591136Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70187 2023-01-11T22:27:39.9591790Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70188 2023-01-11T22:27:39.9592404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:27:39.9592840Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:27:39.9593405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:27:39.9593873Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:27:39.9594488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:27:39.9594939Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:27:39.9595509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:27:39.9595973Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:27:39.9596404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:27:39.9596901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:27:39.9597554Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:27:39.9598236Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:27:39.9598740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:27:39.9599209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:27:39.9599565Z dist init r=1, world=2 2023-01-11T22:27:39.9599796Z dist init r=0, world=2 2023-01-11T22:27:39.9600039Z ok (4.512s) 2023-01-11T22:27:39.9600600Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70270 2023-01-11T22:27:39.9601258Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70271 2023-01-11T22:27:39.9601852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:27:39.9602300Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:27:39.9602874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:27:39.9603326Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:27:39.9603901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:27:39.9604343Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:27:39.9604913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:27:39.9605428Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:27:39.9605876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:27:39.9606364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:27:39.9607024Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:27:39.9607693Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:27:39.9608210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:27:39.9608677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:27:39.9609014Z dist init r=1, world=2 2023-01-11T22:27:39.9609270Z dist init r=0, world=2 2023-01-11T22:27:39.9609509Z ok (4.512s) 2023-01-11T22:27:39.9610070Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70353 2023-01-11T22:27:39.9610884Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70354 2023-01-11T22:27:39.9611581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:27:39.9612042Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:27:39.9612598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:27:39.9613069Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:27:39.9613640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:27:39.9614090Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:27:39.9614641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:27:39.9615105Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:27:39.9615553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:27:39.9616051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:27:39.9616684Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:27:39.9617365Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:27:39.9617883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:27:39.9618335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:27:39.9618690Z dist init r=0, world=2 2023-01-11T22:27:39.9618940Z dist init r=1, world=2 2023-01-11T22:27:39.9619177Z ok (4.412s) 2023-01-11T22:27:39.9619714Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70436 2023-01-11T22:27:39.9620362Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70437 2023-01-11T22:27:39.9620971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:27:39.9621403Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:27:39.9621977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:27:39.9622523Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:27:39.9623105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:27:39.9623527Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:27:39.9624098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:27:39.9624562Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:27:39.9625017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:27:39.9625490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:27:39.9626138Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:27:39.9626821Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:27:39.9627320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:27:39.9627788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:27:39.9628138Z dist init r=1, world=2 2023-01-11T22:27:39.9628466Z dist init r=0, world=2 2023-01-11T22:27:39.9628699Z ok (4.512s) 2023-01-11T22:27:39.9629261Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70519 2023-01-11T22:27:39.9629915Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70520 2023-01-11T22:27:39.9630512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:27:39.9630964Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:27:39.9631532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:27:39.9632004Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:27:39.9632559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:27:39.9633004Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:27:39.9633569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:27:39.9634029Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:27:39.9634460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:27:39.9634959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:27:39.9635611Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:27:39.9636269Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:27:39.9636793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:27:39.9637264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:27:39.9637615Z dist init r=0, world=2 2023-01-11T22:27:39.9637851Z dist init r=1, world=2 2023-01-11T22:27:39.9638087Z ok (4.512s) 2023-01-11T22:27:39.9638641Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70602 2023-01-11T22:27:39.9639342Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70603 2023-01-11T22:27:39.9639948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:27:39.9640395Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:27:39.9640967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:27:39.9641420Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:27:39.9641988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:27:39.9642428Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:27:39.9642999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:27:39.9643453Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:27:39.9643901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:27:39.9644394Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:27:39.9645079Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:27:39.9645783Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:27:39.9646301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:27:39.9646771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:27:39.9647110Z dist init r=0, world=2 2023-01-11T22:27:39.9647363Z dist init r=1, world=2 2023-01-11T22:27:39.9647604Z ok (4.412s) 2023-01-11T22:27:39.9648138Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70685 2023-01-11T22:27:39.9648789Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70686 2023-01-11T22:27:39.9649402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:27:39.9649855Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:27:39.9650576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:27:39.9651053Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:27:39.9651637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:27:39.9652088Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:27:39.9652635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:27:39.9653097Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:27:39.9653547Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:27:39.9654023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:27:39.9654676Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:27:39.9655359Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:27:39.9655882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:27:39.9656425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:27:39.9656775Z dist init r=0, world=2 2023-01-11T22:27:39.9657027Z dist init r=1, world=2 2023-01-11T22:27:39.9657250Z ok (4.512s) 2023-01-11T22:27:39.9657809Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70768 2023-01-11T22:27:39.9658459Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70769 2023-01-11T22:27:39.9659074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:27:39.9659505Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:27:39.9660079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:27:39.9660554Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:27:39.9661133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:27:39.9661557Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:27:39.9662190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:27:39.9662666Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:27:39.9663095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:27:39.9663588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:27:39.9664247Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:27:39.9664942Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:27:39.9665439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:27:39.9665909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:27:39.9666263Z dist init r=0, world=2 2023-01-11T22:27:39.9666500Z dist init r=1, world=2 2023-01-11T22:27:39.9666741Z ok (4.512s) 2023-01-11T22:27:39.9667215Z test_checkpoint_submodule_use_reentrant_False (__main__.TestFSDPCheckpointSubmodule) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70851 2023-01-11T22:27:39.9667787Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70852 2023-01-11T22:27:39.9668377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:27:39.9668833Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:27:39.9669405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:27:39.9669853Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:27:39.9670426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:27:39.9670876Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:27:39.9671442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:27:39.9671884Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:27:39.9672326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:27:39.9672815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:27:39.9673539Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:27:39.9674209Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:27:39.9674725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:27:39.9675199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:27:39.9675538Z dist init r=0, world=2 2023-01-11T22:27:39.9675786Z dist init r=1, world=2 2023-01-11T22:27:39.9676030Z ok (4.413s) 2023-01-11T22:27:39.9676180Z 2023-01-11T22:27:39.9676451Z ---------------------------------------------------------------------- 2023-01-11T22:27:39.9676813Z Ran 17 tests in 78.463s 2023-01-11T22:27:39.9676979Z 2023-01-11T22:27:39.9677072Z OK 2023-01-11T22:27:39.9677212Z 2023-01-11T22:27:39.9677337Z Generating XML reports... 2023-01-11T22:27:39.9677932Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint/TEST-TestFSDPCheckpoint-20230111222621.xml 2023-01-11T22:27:39.9678758Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint/TEST-TestFSDPCheckpointSubmodule-20230111222621.xml 2023-01-11T22:27:39.9679151Z 2023-01-11T22:27:39.9679548Z ##[endgroup] 2023-01-11T22:27:39.9680212Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_checkpoint_o661x9eo) 2023-01-11T22:27:39.9680595Z 2023-01-11T22:27:39.9680849Z Running distributed/test_c10d_pypg ... [2023-01-11 22:27:39.950853] 2023-01-11T22:27:39.9681495Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_pypg.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:27:39.951122] 2023-01-11T22:30:04.4924952Z 2023-01-11T22:30:04.4925560Z Expand the folded group to see the log file of distributed/test_c10d_pypg 2023-01-11T22:30:04.4926445Z ##[group]PRINTING LOG FILE of distributed/test_c10d_pypg (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_pypg_yv0vqqwk) 2023-01-11T22:30:04.4930806Z 2023-01-11T22:30:04.4931458Z Running tests... 2023-01-11T22:30:04.4931992Z ---------------------------------------------------------------------- 2023-01-11T22:30:04.4932580Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_pypg 2023-01-11T22:30:04.4933054Z test_ddp_checkpointing_dynamic_module (__main__.TestDDPWithWorkSubclass) 2023-01-11T22:30:04.4933648Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:30:04.4934145Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70969 2023-01-11T22:30:04.4934752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:30:04.4935221Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:30:04.4935775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:30:04.4936245Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:30:04.4936706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:30:04.4937207Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfs7iuetj 2023-01-11T22:30:04.4937733Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfs7iuetj/_remote_module_non_scriptable.py 2023-01-11T22:30:04.4938125Z ok (5.767s) 2023-01-11T22:30:04.4938478Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.TestDDPWithWorkSubclass) 2023-01-11T22:30:04.4939015Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71006 2023-01-11T22:30:04.4944885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:30:04.4945365Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:30:04.4945963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:30:04.4946431Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:30:04.4946875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:30:04.4947368Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3yjftbr7 2023-01-11T22:30:04.4947907Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3yjftbr7/_remote_module_non_scriptable.py 2023-01-11T22:30:04.4948300Z ok (4.109s) 2023-01-11T22:30:04.4948630Z test_ddp_checkpointing_once_use_reentrant_False (__main__.TestDDPWithWorkSubclass) 2023-01-11T22:30:04.4949192Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71043 2023-01-11T22:30:04.4949882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:30:04.4950342Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:30:04.4951058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:30:04.4951558Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:30:04.4951999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:30:04.4952480Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpes74mj3j 2023-01-11T22:30:04.4953014Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpes74mj3j/_remote_module_non_scriptable.py 2023-01-11T22:30:04.4953529Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.4954096Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.4955264Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2023-01-11T22:30:04.4956084Z warnings.warn( 2023-01-11T22:30:04.4956452Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.4956921Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.4957395Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.4957876Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.4958219Z ok (4.211s) 2023-01-11T22:30:04.4958551Z test_ddp_checkpointing_once_use_reentrant_True (__main__.TestDDPWithWorkSubclass) 2023-01-11T22:30:04.4959094Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71080 2023-01-11T22:30:04.4960064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:30:04.4960510Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:30:04.4961085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:30:04.4961552Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:30:04.4962081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:30:04.4962560Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa1d7jmbt 2023-01-11T22:30:04.4963092Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa1d7jmbt/_remote_module_non_scriptable.py 2023-01-11T22:30:04.4963603Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.4964085Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.4965220Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2023-01-11T22:30:04.4965956Z warnings.warn( 2023-01-11T22:30:04.4966324Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.4966800Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.4967251Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.4967723Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.4968124Z ok (4.310s) 2023-01-11T22:30:04.4968489Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.TestDDPWithWorkSubclass) 2023-01-11T22:30:04.4969191Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71117 2023-01-11T22:30:04.4969882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:30:04.4970738Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:30:04.4971401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:30:04.4971872Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:30:04.4972304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:30:04.4972805Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp36epdpm6 2023-01-11T22:30:04.4973324Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp36epdpm6/_remote_module_non_scriptable.py 2023-01-11T22:30:04.4973833Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.4974309Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.4974639Z ok (4.110s) 2023-01-11T22:30:04.4975012Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.TestDDPWithWorkSubclass) 2023-01-11T22:30:04.4975715Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71154 2023-01-11T22:30:04.4976404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:30:04.4976836Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:30:04.4977403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:30:04.4977867Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:30:04.4978280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:30:04.4978772Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvxza7nhi 2023-01-11T22:30:04.4979306Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvxza7nhi/_remote_module_non_scriptable.py 2023-01-11T22:30:04.4979936Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.4980398Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.4980746Z ok (4.109s) 2023-01-11T22:30:04.4981100Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.TestDDPWithWorkSubclass) 2023-01-11T22:30:04.4981798Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71191 2023-01-11T22:30:04.4982547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:30:04.4982998Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:30:04.4983568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:30:04.4984024Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:30:04.4984456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:30:04.4984947Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5805_4nk 2023-01-11T22:30:04.4985477Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5805_4nk/_remote_module_non_scriptable.py 2023-01-11T22:30:04.4986053Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.4987082Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-01-11T22:30:04.4988095Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.4988447Z ok (4.210s) 2023-01-11T22:30:04.4988798Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.TestDDPWithWorkSubclass) 2023-01-11T22:30:04.4989496Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71228 2023-01-11T22:30:04.4990199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:30:04.4990646Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:30:04.4991213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:30:04.4991665Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:30:04.4992102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:30:04.4992596Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7a370i3o 2023-01-11T22:30:04.4993106Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7a370i3o/_remote_module_non_scriptable.py 2023-01-11T22:30:04.4993482Z ok (4.110s) 2023-01-11T22:30:04.4993834Z test_ddp_checkpointing_twice_weight_sharing (__main__.TestDDPWithWorkSubclass) 2023-01-11T22:30:04.4994392Z Checkpointing should work with static graph in the case of checkpointing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71265 2023-01-11T22:30:04.4995074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:30:04.4995518Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:30:04.4996152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:30:04.4996617Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:30:04.4997030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:30:04.4997526Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8t3hojjx 2023-01-11T22:30:04.4998060Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8t3hojjx/_remote_module_non_scriptable.py 2023-01-11T22:30:04.4998552Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.4999030Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.4999378Z ok (4.109s) 2023-01-11T22:30:04.4999742Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.TestDDPWithWorkSubclass) 2023-01-11T22:30:04.5000303Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71302 2023-01-11T22:30:04.5001000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:30:04.5001448Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:30:04.5002057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:30:04.5002535Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:30:04.5002967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:30:04.5003456Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdzv5s5ul 2023-01-11T22:30:04.5003975Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdzv5s5ul/_remote_module_non_scriptable.py 2023-01-11T22:30:04.5005023Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-01-11T22:30:04.5006694Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2023-01-11T22:30:04.5007424Z warnings.warn( 2023-01-11T22:30:04.5007795Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5008256Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5008603Z ok (4.209s) 2023-01-11T22:30:04.5008964Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.TestDDPWithWorkSubclass) 2023-01-11T22:30:04.5009537Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71339 2023-01-11T22:30:04.5010223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:30:04.5011228Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:30:04.5011809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:30:04.5012259Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:30:04.5012795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:30:04.5013290Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpersc6ppa 2023-01-11T22:30:04.5013835Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpersc6ppa/_remote_module_non_scriptable.py 2023-01-11T22:30:04.5015002Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2023-01-11T22:30:04.5015732Z warnings.warn( 2023-01-11T22:30:04.5016102Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5016581Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5016913Z ok (4.209s) 2023-01-11T22:30:04.5017279Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.TestDDPWithWorkSubclass) 2023-01-11T22:30:04.5017828Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71376 2023-01-11T22:30:04.5018571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:30:04.5019011Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:30:04.5019578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:30:04.5020042Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:30:04.5020457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:30:04.5020952Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo46_j_qi 2023-01-11T22:30:04.5021480Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo46_j_qi/_remote_module_non_scriptable.py 2023-01-11T22:30:04.5021983Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5022442Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5022791Z ok (4.210s) 2023-01-11T22:30:04.5023158Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.TestDDPWithWorkSubclass) 2023-01-11T22:30:04.5023685Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71413 2023-01-11T22:30:04.5024360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:30:04.5024806Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:30:04.5025375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:30:04.5025822Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:30:04.5026251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:30:04.5026751Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiomp5y_7 2023-01-11T22:30:04.5027286Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiomp5y_7/_remote_module_non_scriptable.py 2023-01-11T22:30:04.5027779Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5028256Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5028725Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5029244Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5029586Z ok (4.110s) 2023-01-11T22:30:04.5030021Z test_ddp_invoke_work_object (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71450 2023-01-11T22:30:04.5030714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:30:04.5031146Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:30:04.5031718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:30:04.5032177Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:30:04.5032592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:30:04.5033083Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ait38bm 2023-01-11T22:30:04.5033616Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ait38bm/_remote_module_non_scriptable.py 2023-01-11T22:30:04.5033992Z ok (2.233s) 2023-01-11T22:30:04.5034398Z test_ddp_with_pypg (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71486 2023-01-11T22:30:04.5035084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:30:04.5035581Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:30:04.5036139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:30:04.5036602Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:30:04.5037030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:30:04.5037519Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2y410cbb 2023-01-11T22:30:04.5038038Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2y410cbb/_remote_module_non_scriptable.py 2023-01-11T22:30:04.5038677Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5039028Z ok (2.206s) 2023-01-11T22:30:04.5039474Z test_ddp_with_pypg_with_grad_views (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71522 2023-01-11T22:30:04.5040159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:30:04.5040606Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:30:04.5041172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:30:04.5041619Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:30:04.5042052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:30:04.5042546Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ahqxa_l 2023-01-11T22:30:04.5043079Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ahqxa_l/_remote_module_non_scriptable.py 2023-01-11T22:30:04.5043570Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5043922Z ok (2.206s) 2023-01-11T22:30:04.5044357Z test_invalid_powerSGD_state (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71558 2023-01-11T22:30:04.5045046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:30:04.5045472Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:30:04.5046038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:30:04.5046584Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:30:04.5047006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:30:04.5047788Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2023-01-11T22:30:04.5048861Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2023-01-11T22:30:04.5049931Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2023-01-11T22:30:04.5051580Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2023-01-11T22:30:04.5052655Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2023-01-11T22:30:04.5053706Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2023-01-11T22:30:04.5054334Z ok (2.206s) 2023-01-11T22:30:04.5054761Z test_sync_batch_norm_empty_input (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71592 2023-01-11T22:30:04.5055477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:30:04.5055925Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:30:04.5056475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:30:04.5056943Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:30:04.5057377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:30:04.5057871Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx3i3jppq 2023-01-11T22:30:04.5058391Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx3i3jppq/_remote_module_non_scriptable.py 2023-01-11T22:30:04.5058901Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5059251Z ok (3.609s) 2023-01-11T22:30:04.5059679Z test_sync_batch_norm_only_empty_input (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71629 2023-01-11T22:30:04.5060374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:30:04.5060903Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:30:04.5061469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:30:04.5061920Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:30:04.5062351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:30:04.5062852Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7k8eybo_ 2023-01-11T22:30:04.5063382Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7k8eybo_/_remote_module_non_scriptable.py 2023-01-11T22:30:04.5063875Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5064224Z ok (3.609s) 2023-01-11T22:30:04.5064558Z test_ddp_checkpointing_dynamic_module (__main__.TestDDPWithWorkWrapper) 2023-01-11T22:30:04.5065229Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71666 2023-01-11T22:30:04.5065932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:30:04.5066377Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:30:04.5066994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:30:04.5067455Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:30:04.5067889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:30:04.5068382Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbzlgsk2g 2023-01-11T22:30:04.5068898Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbzlgsk2g/_remote_module_non_scriptable.py 2023-01-11T22:30:04.5069281Z ok (4.210s) 2023-01-11T22:30:04.5069630Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.TestDDPWithWorkWrapper) 2023-01-11T22:30:04.5070188Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71703 2023-01-11T22:30:04.5070866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:30:04.5071314Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:30:04.5071882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:30:04.5072329Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:30:04.5072757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:30:04.5073254Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplkpghxc2 2023-01-11T22:30:04.5073789Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplkpghxc2/_remote_module_non_scriptable.py 2023-01-11T22:30:04.5074154Z ok (4.210s) 2023-01-11T22:30:04.5074504Z test_ddp_checkpointing_once_use_reentrant_False (__main__.TestDDPWithWorkWrapper) 2023-01-11T22:30:04.5075047Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71740 2023-01-11T22:30:04.5075732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:30:04.5076163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:30:04.5076728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:30:04.5077187Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:30:04.5077599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:30:04.5078160Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvorfgmeb 2023-01-11T22:30:04.5078693Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvorfgmeb/_remote_module_non_scriptable.py 2023-01-11T22:30:04.5079204Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5079668Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5080825Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2023-01-11T22:30:04.5081551Z warnings.warn( 2023-01-11T22:30:04.5081922Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5082424Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5082896Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5083368Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5083696Z ok (4.210s) 2023-01-11T22:30:04.5084098Z test_ddp_checkpointing_once_use_reentrant_True (__main__.TestDDPWithWorkWrapper) 2023-01-11T22:30:04.5084645Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71777 2023-01-11T22:30:04.5085331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:30:04.5085761Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:30:04.5086332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:30:04.5086796Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:30:04.5087227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:30:04.5087704Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9z4y2afo 2023-01-11T22:30:04.5088241Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9z4y2afo/_remote_module_non_scriptable.py 2023-01-11T22:30:04.5088747Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5089208Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5090821Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2023-01-11T22:30:04.5091593Z warnings.warn( 2023-01-11T22:30:04.5091964Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5092425Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5092898Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5093370Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5093711Z ok (4.210s) 2023-01-11T22:30:04.5094067Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.TestDDPWithWorkWrapper) 2023-01-11T22:30:04.5094768Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71814 2023-01-11T22:30:04.5095556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:30:04.5096003Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:30:04.5096553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:30:04.5097024Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:30:04.5097457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:30:04.5097935Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzhxy16bu 2023-01-11T22:30:04.5098468Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzhxy16bu/_remote_module_non_scriptable.py 2023-01-11T22:30:04.5098980Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5099456Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5099785Z ok (4.210s) 2023-01-11T22:30:04.5100153Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.TestDDPWithWorkWrapper) 2023-01-11T22:30:04.5100849Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71851 2023-01-11T22:30:04.5101591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:30:04.5102048Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:30:04.5102618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:30:04.5103079Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:30:04.5103494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:30:04.5103991Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpme5l5a7u 2023-01-11T22:30:04.5104523Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpme5l5a7u/_remote_module_non_scriptable.py 2023-01-11T22:30:04.5105014Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5105491Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5105837Z ok (4.109s) 2023-01-11T22:30:04.5106188Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.TestDDPWithWorkWrapper) 2023-01-11T22:30:04.5106877Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71888 2023-01-11T22:30:04.5107577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:30:04.5108024Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:30:04.5108591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:30:04.5109040Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:30:04.5109470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:30:04.5109962Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3iwlk5bt 2023-01-11T22:30:04.5110480Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3iwlk5bt/_remote_module_non_scriptable.py 2023-01-11T22:30:04.5110985Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5111994Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-01-11T22:30:04.5113055Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5113408Z ok (4.111s) 2023-01-11T22:30:04.5113741Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.TestDDPWithWorkWrapper) 2023-01-11T22:30:04.5114443Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71925 2023-01-11T22:30:04.5115142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:30:04.5115576Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:30:04.5116140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:30:04.5116605Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:30:04.5117034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:30:04.5117567Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvcwtw9yv 2023-01-11T22:30:04.5118110Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvcwtw9yv/_remote_module_non_scriptable.py 2023-01-11T22:30:04.5118491Z ok (4.210s) 2023-01-11T22:30:04.5118820Z test_ddp_checkpointing_twice_weight_sharing (__main__.TestDDPWithWorkWrapper) 2023-01-11T22:30:04.5119379Z Checkpointing should work with static graph in the case of checkpointing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71962 2023-01-11T22:30:04.5120080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:30:04.5120528Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:30:04.5121079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:30:04.5121545Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:30:04.5121983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:30:04.5122473Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmputvdk7zo 2023-01-11T22:30:04.5122990Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmputvdk7zo/_remote_module_non_scriptable.py 2023-01-11T22:30:04.5123500Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5123976Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5124312Z ok (4.110s) 2023-01-11T22:30:04.5124672Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.TestDDPWithWorkWrapper) 2023-01-11T22:30:04.5125245Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71999 2023-01-11T22:30:04.5125948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:30:04.5126380Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:30:04.5126945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:30:04.5127405Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:30:04.5127837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:30:04.5128382Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx_e2gp_z 2023-01-11T22:30:04.5128909Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx_e2gp_z/_remote_module_non_scriptable.py 2023-01-11T22:30:04.5129956Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-01-11T22:30:04.5132338Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2023-01-11T22:30:04.5133065Z warnings.warn( 2023-01-11T22:30:04.5133418Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5133898Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5134247Z ok (4.210s) 2023-01-11T22:30:04.5134678Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.TestDDPWithWorkWrapper) 2023-01-11T22:30:04.5135259Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72036 2023-01-11T22:30:04.5135959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:30:04.5136409Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:30:04.5136965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:30:04.5137431Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:30:04.5137865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:30:04.5138365Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6q2mv5hk 2023-01-11T22:30:04.5138880Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6q2mv5hk/_remote_module_non_scriptable.py 2023-01-11T22:30:04.5140062Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2023-01-11T22:30:04.5140790Z warnings.warn( 2023-01-11T22:30:04.5141161Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5141624Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5141984Z ok (4.110s) 2023-01-11T22:30:04.5142354Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.TestDDPWithWorkWrapper) 2023-01-11T22:30:04.5142885Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72073 2023-01-11T22:30:04.5143537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:30:04.5143986Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:30:04.5144553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:30:04.5145088Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:30:04.5145518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:30:04.5146014Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjahfp2gf 2023-01-11T22:30:04.5146552Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjahfp2gf/_remote_module_non_scriptable.py 2023-01-11T22:30:04.5147050Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5147529Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5147876Z ok (4.210s) 2023-01-11T22:30:04.5148220Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.TestDDPWithWorkWrapper) 2023-01-11T22:30:04.5148758Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72110 2023-01-11T22:30:04.5149437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:30:04.5149883Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:30:04.5150428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:30:04.5150947Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:30:04.5151389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:30:04.5151869Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpybu3tbkn 2023-01-11T22:30:04.5152401Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpybu3tbkn/_remote_module_non_scriptable.py 2023-01-11T22:30:04.5152913Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5153395Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5153851Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5154321Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5154666Z ok (4.210s) 2023-01-11T22:30:04.5155085Z test_ddp_invoke_work_object (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72147 2023-01-11T22:30:04.5155783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:30:04.5156231Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:30:04.5156797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:30:04.5157243Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:30:04.5157681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:30:04.5158173Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqn3cby8r 2023-01-11T22:30:04.5158707Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqn3cby8r/_remote_module_non_scriptable.py 2023-01-11T22:30:04.5159069Z ok (2.206s) 2023-01-11T22:30:04.5159498Z test_ddp_with_pypg (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72183 2023-01-11T22:30:04.5160179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:30:04.5160607Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:30:04.5161173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:30:04.5161702Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:30:04.5162135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:30:04.5162614Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi2gcc0vv 2023-01-11T22:30:04.5163145Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi2gcc0vv/_remote_module_non_scriptable.py 2023-01-11T22:30:04.5163656Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5163992Z ok (2.206s) 2023-01-11T22:30:04.5164438Z test_ddp_with_pypg_with_grad_views (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72219 2023-01-11T22:30:04.5165139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:30:04.5165584Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:30:04.5166139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:30:04.5166599Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:30:04.5167027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:30:04.5167504Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7uyxkip9 2023-01-11T22:30:04.5168105Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7uyxkip9/_remote_module_non_scriptable.py 2023-01-11T22:30:04.5168622Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5168970Z ok (2.206s) 2023-01-11T22:30:04.5169391Z test_invalid_powerSGD_state (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72255 2023-01-11T22:30:04.5170086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:30:04.5171019Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:30:04.5171593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:30:04.5172041Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:30:04.5172477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:30:04.5173255Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2023-01-11T22:30:04.5174325Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2023-01-11T22:30:04.5175391Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2023-01-11T22:30:04.5176433Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2023-01-11T22:30:04.5177585Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2023-01-11T22:30:04.5178638Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2023-01-11T22:30:04.5179276Z ok (2.106s) 2023-01-11T22:30:04.5179718Z test_sync_batch_norm_empty_input (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72289 2023-01-11T22:30:04.5180418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:30:04.5180853Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:30:04.5181421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:30:04.5181884Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:30:04.5182344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:30:04.5182916Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpay3il850 2023-01-11T22:30:04.5183460Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpay3il850/_remote_module_non_scriptable.py 2023-01-11T22:30:04.5183972Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5184307Z ok (3.709s) 2023-01-11T22:30:04.5184749Z test_sync_batch_norm_only_empty_input (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72326 2023-01-11T22:30:04.5185452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:30:04.5185880Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:30:04.5186445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:30:04.5186911Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:30:04.5187341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:30:04.5187820Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk7bdzwua 2023-01-11T22:30:04.5188352Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk7bdzwua/_remote_module_non_scriptable.py 2023-01-11T22:30:04.5188861Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:30:04.5189213Z ok (3.609s) 2023-01-11T22:30:04.5189346Z 2023-01-11T22:30:04.5189618Z ---------------------------------------------------------------------- 2023-01-11T22:30:04.5189946Z Ran 38 tests in 142.227s 2023-01-11T22:30:04.5190108Z 2023-01-11T22:30:04.5190199Z OK 2023-01-11T22:30:04.5190315Z 2023-01-11T22:30:04.5190437Z Generating XML reports... 2023-01-11T22:30:04.5191034Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_pypg/TEST-TestDDPWithWorkSubclass-20230111222741.xml 2023-01-11T22:30:04.5191805Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_pypg/TEST-TestDDPWithWorkWrapper-20230111222741.xml 2023-01-11T22:30:04.5192155Z 2023-01-11T22:30:04.5192519Z ##[endgroup] 2023-01-11T22:30:04.5193072Z FINISHED PRINTING LOG FILE of distributed/test_c10d_pypg (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_pypg_yv0vqqwk) 2023-01-11T22:30:04.5193400Z 2023-01-11T22:30:04.5193750Z Running distributed/fsdp/test_fsdp_use_orig_params ... [2023-01-11 22:30:04.493078] 2023-01-11T22:30:04.5194448Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_use_orig_params.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:30:04.493354] 2023-01-11T22:33:13.2361684Z 2023-01-11T22:33:13.2364338Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_use_orig_params 2023-01-11T22:33:13.2365349Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_use_orig_params (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_use_orig_params_pv8y0c30) 2023-01-11T22:33:13.2365733Z 2023-01-11T22:33:13.2365847Z Running tests... 2023-01-11T22:33:13.2371739Z ---------------------------------------------------------------------- 2023-01-11T22:33:13.2372328Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params 2023-01-11T22:33:13.2372834Z test_named_parameters_in_forward (__main__.TestFSDPUseOrigParamsFQNs) 2023-01-11T22:33:13.2373343Z Tests that calling ``named_parameters()`` during forward returns FQNs ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T22:33:13.2373841Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72398 2023-01-11T22:33:13.2374273Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72399 2023-01-11T22:33:13.2375189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:33:13.2375674Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:33:13.2376273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:33:13.2379155Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:33:13.2380502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:33:13.2381261Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:33:13.2381876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:33:13.2382346Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:33:13.2382784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:33:13.2383289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:33:13.2383948Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:33:13.2384624Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:33:13.2385150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:33:13.2385622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:33:13.2385980Z dist init r=1, world=2 2023-01-11T22:33:13.2386218Z dist init r=0, world=2 2023-01-11T22:33:13.2386471Z ok (6.029s) 2023-01-11T22:33:13.2386961Z test_param_and_buffer_names (__main__.TestFSDPUseOrigParamsFQNs) 2023-01-11T22:33:13.2388015Z Tests that, for ``use_orig_params=True``, the parameter and buffer ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72477 2023-01-11T22:33:13.2389016Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72478 2023-01-11T22:33:13.2390215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:33:13.2391022Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:33:13.2392097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:33:13.2393261Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:33:13.2394350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:33:13.2395197Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:33:13.2396229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:33:13.2397053Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:33:13.2398427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:33:13.2399316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:33:13.2400538Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:33:13.2401755Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:33:13.2402639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:33:13.2403437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:33:13.2405839Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:33:13.2407345Z warnings.warn( 2023-01-11T22:33:13.2409646Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:33:13.2410696Z warnings.warn( 2023-01-11T22:33:13.2410934Z dist init r=0, world=2 2023-01-11T22:33:13.2411189Z dist init r=1, world=2 2023-01-11T22:33:13.2411424Z ok (4.010s) 2023-01-11T22:33:13.2411841Z test_diff_hyperparams_cpu_offload_sharding_strategy_str_full_shard (__main__.TestFSDPUseOrigParamsMultipleParamGroups) 2023-01-11T22:33:13.2412477Z Tests FSDP parity with DDP when using multiple parameter groups with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72556 2023-01-11T22:33:13.2413013Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72557 2023-01-11T22:33:13.2413640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:33:13.2414068Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:33:13.2414636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:33:13.2415100Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:33:13.2415660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:33:13.2416100Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:33:13.2416662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:33:13.2417121Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:33:13.2417552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:33:13.2418171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:33:13.2418830Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:33:13.2419515Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:33:13.2420018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:33:13.2420487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:33:13.2420963Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2421426Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2421785Z dist init r=0, world=2 2023-01-11T22:33:13.2422036Z dist init r=1, world=2 2023-01-11T22:33:13.2422254Z ok (5.413s) 2023-01-11T22:33:13.2422682Z test_diff_hyperparams_cpu_offload_sharding_strategy_str_no_shard (__main__.TestFSDPUseOrigParamsMultipleParamGroups) 2023-01-11T22:33:13.2423308Z Tests FSDP parity with DDP when using multiple parameter groups with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72639 2023-01-11T22:33:13.2423924Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72640 2023-01-11T22:33:13.2424534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:33:13.2424983Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:33:13.2425547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:33:13.2426015Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:33:13.2426572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:33:13.2427008Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:33:13.2427569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:33:13.2428016Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:33:13.2428465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:33:13.2428958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:33:13.2429604Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:33:13.2430261Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:33:13.2430777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:33:13.2431238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:33:13.2431709Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2432172Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2432535Z dist init r=0, world=2 2023-01-11T22:33:13.2432782Z dist init r=1, world=2 2023-01-11T22:33:13.2433003Z ok (5.415s) 2023-01-11T22:33:13.2433437Z test_diff_hyperparams_cpu_offload_sharding_strategy_str_shard_grad_op (__main__.TestFSDPUseOrigParamsMultipleParamGroups) 2023-01-11T22:33:13.2451202Z Tests FSDP parity with DDP when using multiple parameter groups with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72722 2023-01-11T22:33:13.2451966Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72723 2023-01-11T22:33:13.2452623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:33:13.2453081Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:33:13.2453656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:33:13.2454116Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:33:13.2454686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:33:13.2455127Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:33:13.2455690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:33:13.2456132Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:33:13.2456582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:33:13.2457075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:33:13.2457709Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:33:13.2458466Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:33:13.2459005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:33:13.2459467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:33:13.2459925Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2460404Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2460770Z dist init r=1, world=2 2023-01-11T22:33:13.2461002Z dist init r=0, world=2 2023-01-11T22:33:13.2461242Z ok (5.413s) 2023-01-11T22:33:13.2461660Z test_diff_hyperparams_sharding_strategy_str_full_shard (__main__.TestFSDPUseOrigParamsMultipleParamGroups) 2023-01-11T22:33:13.2462278Z Tests FSDP parity with DDP when using multiple parameter groups with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72805 2023-01-11T22:33:13.2462803Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72806 2023-01-11T22:33:13.2463415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:33:13.2463862Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:33:13.2464413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:33:13.2464885Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:33:13.2465458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:33:13.2465898Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:33:13.2466443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:33:13.2466909Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:33:13.2467356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:33:13.2467849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:33:13.2468486Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:33:13.2469165Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:33:13.2469738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:33:13.2470187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:33:13.2470659Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2471143Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2471617Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2472071Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2472538Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2473004Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2473477Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2473930Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2474394Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2474860Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2475357Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2475832Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2476295Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2476755Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2477207Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2477678Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2478142Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2478589Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2479053Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2479517Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2479982Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2480430Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2480894Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2481359Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2481806Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2482272Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2482735Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2483200Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2483643Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2484106Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2484565Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2485011Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2485534Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2485994Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2486458Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2486904Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2487370Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2487830Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2488289Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2488738Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2489199Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2489665Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2490165Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2490895Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2491362Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2491898Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2492361Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2492822Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2494089Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:33:13.2494887Z warnings.warn( 2023-01-11T22:33:13.2496008Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:33:13.2496780Z warnings.warn( 2023-01-11T22:33:13.2497144Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2497618Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2498072Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2498546Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2499012Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2499478Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2499924Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2500388Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2500853Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2501301Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2501867Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2502331Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2502796Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2503246Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2503712Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2504170Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2504635Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2505084Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2505546Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2506008Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2506452Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2506912Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2507375Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2507888Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2508344Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2508807Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2509270Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2509724Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2510186Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2510645Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2511105Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2511555Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2512015Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2512476Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2512917Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2513374Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2513840Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2514301Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2514749Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2515211Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2515670Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2516109Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2516570Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2517019Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2517459Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2517955Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2518403Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2518867Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2519217Z dist init r=0, world=2 2023-01-11T22:33:13.2519450Z dist init r=1, world=2 2023-01-11T22:33:13.2519686Z ok (33.656s) 2023-01-11T22:33:13.2520108Z test_diff_hyperparams_sharding_strategy_str_no_shard (__main__.TestFSDPUseOrigParamsMultipleParamGroups) 2023-01-11T22:33:13.2520707Z Tests FSDP parity with DDP when using multiple parameter groups with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72888 2023-01-11T22:33:13.2521246Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72889 2023-01-11T22:33:13.2521861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:33:13.2522299Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:33:13.2522866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:33:13.2523336Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:33:13.2523960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:33:13.2524395Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:33:13.2524959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:33:13.2525423Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:33:13.2525870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:33:13.2526348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:33:13.2526994Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:33:13.2527673Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:33:13.2528174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:33:13.2528639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:33:13.2529106Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2529585Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2530864Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2532139Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2533367Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2534574Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2535894Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2537119Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2538342Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2539625Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2540847Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2542060Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2542789Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2543272Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2543746Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2544206Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2545184Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2546406Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2547623Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2548831Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2550123Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2551342Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2552545Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2553765Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2555024Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2556244Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2557458Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2558673Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2559889Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2561104Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2562305Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2563514Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2564786Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2565998Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2567208Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2568420Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2569145Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2569725Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2570214Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2570958Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2571940Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2573161Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2574372Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2575586Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2576808Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2578027Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2579243Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2580540Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2581761Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2582963Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2584167Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2585439Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2586671Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2587879Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2589095Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2590339Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2591551Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2592755Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2593971Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2595225Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2595950Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2596433Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2596908Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2597365Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2598342Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2599609Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2600840Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2602055Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2603279Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2604487Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2605704Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2606907Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2608119Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2609323Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2610866Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2612095Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2613305Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2614578Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2615808Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2616997Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2618212Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2619421Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2620632Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2621839Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2622567Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2623053Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2623510Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2623983Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2624954Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2626248Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2627464Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2628653Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2629921Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2631136Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2632341Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2633553Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2634767Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2635975Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2637194Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2638388Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2639604Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2640871Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2642081Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2643286Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2644550Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2645771Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2646984Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2648202Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2648910Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2649392Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2649865Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2650513Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2651495Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2652716Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2653931Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2655223Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2656449Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2657662Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2658881Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2660137Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2661376Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2662586Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2663798Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2665008Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2666221Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2667434Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2668647Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2669915Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2671341Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2672555Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2673770Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2675031Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2675765Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2676247Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2676704Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2677183Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2678217Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2679444Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2680666Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2681865Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2683091Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2684304Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2685584Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2686800Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2688010Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2689224Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2690758Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2692000Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2693197Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2694407Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2695624Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2696830Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2698040Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2699244Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2700530Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2701746Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2702456Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2702936Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2703412Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2703884Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2704890Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2706116Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2707332Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2708556Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2709770Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2710983Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2712200Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2713413Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2714607Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2715887Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2716612Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2717091Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2717562Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2718021Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2718998Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2720258Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2721489Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2722705Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2723923Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2725131Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2726335Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2727545Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2728761Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2730024Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2731413Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2732622Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2733839Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2735104Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2736318Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2737532Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2738755Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2739966Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2741180Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2742388Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2743112Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2743576Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2744048Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2744597Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2745576Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2746798Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2748001Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2749208Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2750470Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2751694Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2752910Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2754124Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2755335Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2756545Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2757735Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2758949Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2760219Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2761433Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2762640Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2763847Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2765103Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2766322Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2767542Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2768734Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2769455Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2769938Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2770614Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2771079Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2772059Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2773281Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2774500Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2775789Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2777011Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2778226Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2779485Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2780707Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2781925Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2783133Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2784342Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2785551Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2786774Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2787985Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2789196Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2790490Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2791698Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2792901Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2794165Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2795379Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2796102Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2796590Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2797045Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2797522Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2798497Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2799709Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2800915Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2802132Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2803352Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2804620Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2805836Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2807049Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2808270Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2809523Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2811341Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2812580Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2813802Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2815016Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2816230Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2817436Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2818647Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2819977Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2821189Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2822396Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2823107Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2823593Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2824955Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:33:13.2825743Z warnings.warn( 2023-01-11T22:33:13.2826871Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:33:13.2827643Z warnings.warn( 2023-01-11T22:33:13.2828011Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2828493Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2829485Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2830693Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2831914Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2833129Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2834357Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2835627Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2836853Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2838064Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2839278Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2840527Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2841742Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2842964Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2844181Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2845387Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2846604Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2847815Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2849031Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2850537Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2851776Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2852996Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2853727Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2854211Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2854684Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2855219Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2856209Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2857425Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2858652Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2859863Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2861082Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2862293Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2863488Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2864701Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2866013Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2867217Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2868426Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2869680Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2870905Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2872109Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2873334Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2874532Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2875743Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2876956Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2878172Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2879382Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2880165Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2880629Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2881106Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2881585Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2882560Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2883776Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2885031Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2886258Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2887488Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2888696Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2889943Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2891361Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2892583Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2893797Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2895097Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2896294Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2897509Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2898718Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2899990Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2901209Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2902435Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2903640Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2904855Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2906044Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2906773Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2907260Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2907734Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2908196Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2909175Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2910449Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2911667Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2912883Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2914109Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2915365Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2916592Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2917793Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2919005Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2920218Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2921433Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2922645Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2923858Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2925133Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2926346Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2927537Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2928753Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2930004Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2931401Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2932615Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2933340Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2933824Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2934282Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2934758Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2935728Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2936946Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2938167Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2939366Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2940673Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2941886Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2943099Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2944363Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2945596Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2946802Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2948020Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2949202Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2950415Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2951886Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2953096Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2954301Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2955576Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2956788Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2957997Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2959212Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2959959Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2960447Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2960923Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2961397Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2962357Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2963573Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2964792Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2966007Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2967223Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2968435Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2969650Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2971142Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2972351Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2973564Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2974852Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2976069Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2977275Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2978533Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2979758Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2980966Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2982188Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2983378Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2984577Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2985853Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2986578Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2987055Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2987506Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2987978Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.2988949Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2990233Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2991458Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2992671Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2993882Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2995090Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2996312Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2997517Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2998729Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.2999992Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3001212Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3002424Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3003642Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3004883Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3006110Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3007323Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3008535Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3009738Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3011126Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3012345Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3013072Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3013537Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3014015Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3014574Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3015535Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3016751Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3017976Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3019186Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3020461Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3021683Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3022909Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3024120Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3025337Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3026531Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3027750Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3028959Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3030244Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3031457Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3032678Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3033891Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3035147Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3036349Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3037566Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3038780Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3039510Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3039993Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3040472Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3040934Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3041914Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3043131Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3044355Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3045636Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3046854Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3048046Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3049314Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3050757Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3051990Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3053202Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3054422Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3055626Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3056848Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3058033Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3059253Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3060607Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3061818Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3063016Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3064282Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3065505Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3066227Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3066696Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3067167Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3067645Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3068623Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3069811Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3071037Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3072251Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3073466Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3074732Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3075950Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3077163Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3078380Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3079617Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3080844Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3082060Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3083275Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3084474Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3085683Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3086892Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3088106Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3089370Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3090804Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3092029Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3092752Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3093233Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3093704Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3094163Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3095207Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3096442Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3097663Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3098872Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3100089Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3101299Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3102505Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3103716Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3104998Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3106204Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3107415Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3108632Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3109889Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3111112Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3112309Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3113524Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3114738Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3115941Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3117155Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3118361Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3119635Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3120842Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3122052Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3123244Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3124503Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3125729Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3126940Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3128148Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3129363Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3130779Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3131519Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3131983Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3132456Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3132933Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3134163Z /opt/conda/lib/python3.10/site-packages/torch/_tensor.py:795: UserWarning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:319.) 2023-01-11T22:33:13.3135083Z return torch._VF.split_with_sizes(self, split_size, dim) 2023-01-11T22:33:13.3136247Z /opt/conda/lib/python3.10/site-packages/torch/_tensor.py:795: UserWarning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:319.) 2023-01-11T22:33:13.3137060Z return torch._VF.split_with_sizes(self, split_size, dim) 2023-01-11T22:33:13.3137477Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3137942Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3138302Z dist init r=0, world=2 2023-01-11T22:33:13.3138550Z dist init r=1, world=2 2023-01-11T22:33:13.3138788Z ok (34.359s) 2023-01-11T22:33:13.3139195Z test_diff_hyperparams_sharding_strategy_str_shard_grad_op (__main__.TestFSDPUseOrigParamsMultipleParamGroups) 2023-01-11T22:33:13.3139882Z Tests FSDP parity with DDP when using multiple parameter groups with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72971 2023-01-11T22:33:13.3140431Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72972 2023-01-11T22:33:13.3141029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:33:13.3141479Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:33:13.3142049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:33:13.3142520Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:33:13.3143075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:33:13.3143520Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:33:13.3144084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:33:13.3144529Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:33:13.3144974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:33:13.3145473Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:33:13.3146124Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:33:13.3146798Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:33:13.3147319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:33:13.3147786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:33:13.3148263Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3148723Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3149198Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3149670Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3150121Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3150659Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3151129Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3151594Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3152045Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3152516Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3152982Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3153431Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3153899Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3154362Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3154832Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3155283Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3155744Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3156274Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3156750Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3157202Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3157671Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3158134Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3158588Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3159052Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3159516Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3159981Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3160431Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3160893Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3161354Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3161800Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3162267Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3162736Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3163203Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3163647Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3164107Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3164573Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3165018Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3165485Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3165946Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3166407Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3166916Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3167378Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3167845Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3168288Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3168757Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3169222Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3169686Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3170135Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3171620Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:33:13.3172419Z warnings.warn( 2023-01-11T22:33:13.3173648Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2023-01-11T22:33:13.3174453Z warnings.warn( 2023-01-11T22:33:13.3174805Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3175280Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3175750Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3176222Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3176677Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3177141Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3177609Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3178061Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3178532Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3179000Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3179461Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3179912Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3180379Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3180849Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3181295Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3181758Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3182222Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3182686Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3183215Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3183678Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3184139Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3184610Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3185059Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3185527Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3185992Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3186437Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3186903Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3187371Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3187838Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3188292Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3188801Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3189283Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3189731Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3190237Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3190702Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3191165Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3191614Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3192078Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3192542Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3192992Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3193458Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3193925Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3194385Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3194835Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3195299Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3195758Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3196205Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3196670Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3197026Z dist init r=1, world=2 2023-01-11T22:33:13.3197280Z dist init r=0, world=2 2023-01-11T22:33:13.3197499Z ok (33.059s) 2023-01-11T22:33:13.3197871Z test_diff_trainability (__main__.TestFSDPUseOrigParamsMultipleParamGroups) 2023-01-11T22:33:13.3198452Z Tests FSDP parity with DDP when using multiple parameter groups and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73054 2023-01-11T22:33:13.3198974Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73055 2023-01-11T22:33:13.3199647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:33:13.3200100Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:33:13.3200671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:33:13.3201123Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:33:13.3201697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:33:13.3202141Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:33:13.3202689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:33:13.3203149Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:33:13.3203605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:33:13.3204101Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:33:13.3204737Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:33:13.3205470Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:33:13.3205995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:33:13.3206456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:33:13.3207432Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3208678Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3209905Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3211356Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3212574Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3213803Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3215024Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3216340Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3217545Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3218769Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3219378Z dist init r=1, world=2 2023-01-11T22:33:13.3219636Z dist init r=0, world=2 2023-01-11T22:33:13.3219857Z ok (8.223s) 2023-01-11T22:33:13.3220237Z test_multiple_optimizers (__main__.TestFSDPUseOrigParamsMultipleParamGroups) 2023-01-11T22:33:13.3220810Z Tests using two optimizers where only one sets gradients to ``None``. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73137 2023-01-11T22:33:13.3221398Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73138 2023-01-11T22:33:13.3222002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:33:13.3222452Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:33:13.3223021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:33:13.3223498Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:33:13.3224052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:33:13.3224494Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:33:13.3225059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:33:13.3225508Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:33:13.3225959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:33:13.3226454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:33:13.3227106Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:33:13.3227775Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:33:13.3228296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:33:13.3228762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:33:13.3229773Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-01-11T22:33:13.3231312Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-01-11T22:33:13.3232242Z dist init r=1, world=2 2023-01-11T22:33:13.3232479Z dist init r=0, world=2 2023-01-11T22:33:13.3232717Z ok (5.515s) 2023-01-11T22:33:13.3233026Z test_no_sync (__main__.TestFSDPUseOrigParamsNoSync) 2023-01-11T22:33:13.3233539Z Tests a basic ``no_sync()`` setup by comparing ``use_orig_params=True`` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73220 2023-01-11T22:33:13.3234053Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73221 2023-01-11T22:33:13.3234664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:33:13.3235120Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:33:13.3235673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:33:13.3236139Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:33:13.3236761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:33:13.3237212Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:33:13.3237762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:33:13.3238224Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:33:13.3238672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:33:13.3239171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:33:13.3239805Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:33:13.3240485Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:33:13.3241009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:33:13.3241458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:33:13.3241811Z dist init r=1, world=2 2023-01-11T22:33:13.3242063Z dist init r=0, world=2 2023-01-11T22:33:13.3242301Z ok (4.512s) 2023-01-11T22:33:13.3242638Z test_access_params_after_forward (__main__.TestFSDPUseOrigParamsParamAccess) 2023-01-11T22:33:13.3243200Z Tests that accessing the original parameters after the forward but ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73303 2023-01-11T22:33:13.3243736Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73304 2023-01-11T22:33:13.3244327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:33:13.3244774Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:33:13.3245348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:33:13.3245815Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:33:13.3246372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:33:13.3246814Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:33:13.3247376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:33:13.3247897Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:33:13.3248344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:33:13.3248839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:33:13.3249496Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:33:13.3250167Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:33:13.3250900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:33:13.3251372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:33:13.3251846Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3252313Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3252793Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3253265Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3253792Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3254284Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3254397Z dist init r=0, world=2 2023-01-11T22:33:13.3254508Z dist init r=1, world=2 2023-01-11T22:33:13.3254608Z ok (4.511s) 2023-01-11T22:33:13.3254862Z test_multiple_forward_offload_params_False (__main__.TestFSDPUseOrigParamsUnshardReshard) 2023-01-11T22:33:13.3255143Z Tests that ``use_orig_params=True`` has parity with ``False`` when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73386 2023-01-11T22:33:13.3255362Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73387 2023-01-11T22:33:13.3255743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:33:13.3255916Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:33:13.3256299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:33:13.3256486Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:33:13.3256846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:33:13.3257017Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:33:13.3257387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:33:13.3257559Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:33:13.3257799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:33:13.3258037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:33:13.3258433Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:33:13.3258827Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:33:13.3259054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:33:13.3259276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:33:13.3260022Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3260827Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3261562Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3262277Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3263057Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3263790Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3264523Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3265248Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3265973Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3266687Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3267418Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3268134Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3268857Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3269629Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3269744Z dist init r=0, world=2 2023-01-11T22:33:13.3269838Z dist init r=1, world=2 2023-01-11T22:33:13.3269939Z ok (5.713s) 2023-01-11T22:33:13.3270196Z test_multiple_forward_offload_params_True (__main__.TestFSDPUseOrigParamsUnshardReshard) 2023-01-11T22:33:13.3270493Z Tests that ``use_orig_params=True`` has parity with ``False`` when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73469 2023-01-11T22:33:13.3270711Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73470 2023-01-11T22:33:13.3271080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:33:13.3271252Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:33:13.3271628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:33:13.3271845Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:33:13.3272216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:33:13.3272386Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:33:13.3272751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:33:13.3272941Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:33:13.3273182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:33:13.3273421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:33:13.3273815Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:33:13.3274189Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:33:13.3274415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:33:13.3274635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:33:13.3275371Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3276096Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3276825Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3277538Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3278380Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3279101Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3279826Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3280548Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3281363Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3282087Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3282815Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3283530Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3284250Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3284964Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3285082Z dist init r=1, world=2 2023-01-11T22:33:13.3285190Z dist init r=0, world=2 2023-01-11T22:33:13.3285288Z ok (5.813s) 2023-01-11T22:33:13.3285560Z test_summon_between_two_forwards_offload_params_False (__main__.TestFSDPUseOrigParamsUnshardReshard) 2023-01-11T22:33:13.3285844Z Tests that ``use_orig_params=True`` has parity with ``False`` when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73552 2023-01-11T22:33:13.3286060Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73553 2023-01-11T22:33:13.3286498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:33:13.3286670Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:33:13.3287047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:33:13.3287238Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:33:13.3287597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:33:13.3287770Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:33:13.3288139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:33:13.3288308Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:33:13.3288552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:33:13.3288791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:33:13.3289187Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:33:13.3289623Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:33:13.3289858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:33:13.3290120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:33:13.3291081Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3291825Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3292558Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3293278Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3294012Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3294736Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3295462Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3296266Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3296992Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3297702Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3298424Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3299192Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3299933Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3300649Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3300747Z dist init r=1, world=2 2023-01-11T22:33:13.3300860Z dist init r=0, world=2 2023-01-11T22:33:13.3300961Z ok (6.114s) 2023-01-11T22:33:13.3301228Z test_summon_between_two_forwards_offload_params_True (__main__.TestFSDPUseOrigParamsUnshardReshard) 2023-01-11T22:33:13.3301528Z Tests that ``use_orig_params=True`` has parity with ``False`` when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73635 2023-01-11T22:33:13.3301744Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73636 2023-01-11T22:33:13.3302117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:33:13.3302289Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:33:13.3302647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:33:13.3302837Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:33:13.3303197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:33:13.3303366Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:33:13.3303731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:33:13.3303916Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:33:13.3304157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:33:13.3304455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:33:13.3304853Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:33:13.3305226Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:33:13.3305452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:33:13.3305677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:33:13.3306417Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3307135Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3307909Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3308637Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3309370Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3310083Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3310812Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3311530Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3312256Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3312968Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3313751Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3314463Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3315183Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3315893Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:33:13.3316004Z dist init r=1, world=2 2023-01-11T22:33:13.3316113Z dist init r=0, world=2 2023-01-11T22:33:13.3316255Z ok (6.214s) 2023-01-11T22:33:13.3316448Z test_grad_writeback (__main__.TestFSDPUseOrigParamsWriteback) 2023-01-11T22:33:13.3316897Z Tests that changes to the original parameters' gradients are written ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73718 2023-01-11T22:33:13.3317114Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73719 2023-01-11T22:33:13.3317478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:33:13.3317653Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:33:13.3318029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:33:13.3318219Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:33:13.3318581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:33:13.3318751Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:33:13.3319106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:33:13.3319293Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:33:13.3319533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:33:13.3319776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:33:13.3320170Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:33:13.3320559Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:33:13.3320785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:33:13.3321009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:33:13.3321241Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3321455Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3321686Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3321964Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3322192Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3322415Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3322638Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3322860Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3323078Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3323276Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3323501Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3323722Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2023-01-11T22:33:13.3323837Z dist init r=0, world=2 2023-01-11T22:33:13.3323944Z dist init r=1, world=2 2023-01-11T22:33:13.3324044Z ok (4.511s) 2023-01-11T22:33:13.3324249Z test_param_writeback (__main__.TestFSDPUseOrigParamsWriteback) 2023-01-11T22:33:13.3324557Z Tests that changes to the original parameters are written back. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73801 2023-01-11T22:33:13.3324806Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73802 2023-01-11T22:33:13.3325191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:33:13.3325364Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:33:13.3325740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:33:13.3325925Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:33:13.3326288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:33:13.3326458Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:33:13.3326833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:33:13.3327016Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:33:13.3327244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:33:13.3327481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:33:13.3327877Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:33:13.3328266Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:33:13.3328495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:33:13.3328715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:33:13.3328826Z dist init r=1, world=2 2023-01-11T22:33:13.3328933Z dist init r=0, world=2 2023-01-11T22:33:13.3329016Z ok (4.010s) 2023-01-11T22:33:13.3329356Z test_writeback_shape_mismatch (__main__.TestFSDPUseOrigParamsWriteback) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73880 2023-01-11T22:33:13.3329571Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73881 2023-01-11T22:33:13.3329940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:33:13.3330113Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:33:13.3330703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:33:13.3330978Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:33:13.3331346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:33:13.3331500Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:33:13.3331869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:33:13.3332052Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:33:13.3332292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:33:13.3332529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:33:13.3332921Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:33:13.3333316Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:33:13.3333540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:33:13.3333761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:33:13.3333855Z dist init r=1, world=2 2023-01-11T22:33:13.3334022Z dist init r=0, world=2 2023-01-11T22:33:13.3334131Z ok (3.911s) 2023-01-11T22:33:13.3334153Z 2023-01-11T22:33:13.3334422Z ---------------------------------------------------------------------- 2023-01-11T22:33:13.3334539Z Ran 19 tests in 186.402s 2023-01-11T22:33:13.3334559Z 2023-01-11T22:33:13.3334651Z OK 2023-01-11T22:33:13.3334670Z 2023-01-11T22:33:13.3334792Z Generating XML reports... 2023-01-11T22:33:13.3335283Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsFQNs-20230111223006.xml 2023-01-11T22:33:13.3335820Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsMultipleParamGroups-20230111223006.xml 2023-01-11T22:33:13.3336306Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsNoSync-20230111223006.xml 2023-01-11T22:33:13.3336811Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsParamAccess-20230111223006.xml 2023-01-11T22:33:13.3337326Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsUnshardReshard-20230111223006.xml 2023-01-11T22:33:13.3337820Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsWriteback-20230111223006.xml 2023-01-11T22:33:13.3337840Z 2023-01-11T22:33:13.3338297Z ##[endgroup] 2023-01-11T22:33:13.3338796Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_use_orig_params (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_use_orig_params_pv8y0c30) 2023-01-11T22:33:13.3338816Z 2023-01-11T22:33:13.3339098Z Running distributed/fsdp/test_fsdp_mixed_precision ... [2023-01-11 22:33:13.238569] 2023-01-11T22:33:13.3339586Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_mixed_precision.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:33:13.238860] 2023-01-11T22:39:19.0563616Z 2023-01-11T22:39:19.0566990Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_mixed_precision 2023-01-11T22:39:19.0568121Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_mixed_precision (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_mixed_precision_r7m2hb6r) 2023-01-11T22:39:19.0568646Z 2023-01-11T22:39:19.0568797Z Running tests... 2023-01-11T22:39:19.0569779Z ---------------------------------------------------------------------- 2023-01-11T22:39:19.0570895Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision 2023-01-11T22:39:19.0571569Z test_float16_on_one_submodule (__main__.TestFSDPDifferentSubmodulePrecision) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73994 2023-01-11T22:39:19.0572700Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73995 2023-01-11T22:39:19.0573485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.0574018Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.0574616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.0575267Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.0575980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.0576505Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.0577094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.0578616Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.0579332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.0579873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.0581181Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.0586490Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.0587751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.0588622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.0589068Z dist init r=1, world=2 2023-01-11T22:39:19.0589959Z dist init r=0, world=2 2023-01-11T22:39:19.0590210Z ok (6.028s) 2023-01-11T22:39:19.0590823Z test_float16_on_one_submodule_skip_inputs (__main__.TestFSDPDifferentSubmodulePrecision) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74078 2023-01-11T22:39:19.0591615Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74079 2023-01-11T22:39:19.0592267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.0592811Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.0593484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.0594047Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.0621074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.0621960Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.0622687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.0623158Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.0623601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.0624103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.0624774Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.0625663Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.0626179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.0626629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.0626991Z dist init r=1, world=2 2023-01-11T22:39:19.0627237Z dist init r=0, world=2 2023-01-11T22:39:19.0627464Z ok (5.914s) 2023-01-11T22:39:19.0627967Z test_float16_on_one_submodule_skip_inputs_error (__main__.TestFSDPDifferentSubmodulePrecision) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74162 2023-01-11T22:39:19.0628558Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74163 2023-01-11T22:39:19.0629185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.0629888Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.0630456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.0630933Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.0631501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.0632022Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.0632621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.0633085Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.0633519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.0634029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.0634664Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.0635377Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.0635871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.0636345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.0636707Z dist init r=1, world=2 2023-01-11T22:39:19.0636967Z dist init r=0, world=2 2023-01-11T22:39:19.0637179Z ok (5.915s) 2023-01-11T22:39:19.0637689Z test_submodules_with_different_precisions (__main__.TestFSDPDifferentSubmodulePrecision) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74242 2023-01-11T22:39:19.0638272Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74243 2023-01-11T22:39:19.0638878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.0639319Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.0639899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.0640382Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.0640937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.0641374Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.0641954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.0642399Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.0642928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.0643419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.0644075Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.0644754Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.0645257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.0645734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.0646089Z dist init r=1, world=2 2023-01-11T22:39:19.0646324Z dist init r=0, world=2 2023-01-11T22:39:19.0646572Z ok (5.914s) 2023-01-11T22:39:19.0647077Z test_submodules_with_different_precisions_error (__main__.TestFSDPDifferentSubmodulePrecision) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74326 2023-01-11T22:39:19.0647685Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74327 2023-01-11T22:39:19.0648267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.0648723Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.0649356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.0649832Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.0650945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.0651409Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.0652003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.0652423Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.0652904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.0653402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.0654064Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.0654729Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.0655247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.0655712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.0656062Z dist init r=0, world=2 2023-01-11T22:39:19.0656315Z dist init r=1, world=2 2023-01-11T22:39:19.0656556Z ok (5.914s) 2023-01-11T22:39:19.0657015Z test_submodules_with_external_inputs (__main__.TestFSDPDifferentSubmodulePrecision) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74406 2023-01-11T22:39:19.0657595Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74407 2023-01-11T22:39:19.0658223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.0658673Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.0659222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.0659693Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.0660236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.0660793Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.0661327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.0661789Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.0662243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.0662721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.0663396Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.0664054Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.0664571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.0665025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.0665372Z dist init r=0, world=2 2023-01-11T22:39:19.0665626Z dist init r=1, world=2 2023-01-11T22:39:19.0665848Z ok (5.915s) 2023-01-11T22:39:19.0666411Z test_mixed_precision_with_ignored_module (__main__.TestFSDPMixedPrecisionIgnoredModules) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74490 2023-01-11T22:39:19.0667166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.0667630Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.0668167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.0668631Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.0669087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.0669739Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2023-01-11T22:39:19.0670245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.0671016Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:288: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1. 2023-01-11T22:39:19.0671509Z warnings.warn( 2023-01-11T22:39:19.0671740Z dist init r=0, world=1 2023-01-11T22:39:19.0671981Z ok (4.510s) 2023-01-11T22:39:19.0672444Z test_grads_reduced_precision (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74530 2023-01-11T22:39:19.0673005Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74531 2023-01-11T22:39:19.0673595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.0674043Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.0674614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.0675069Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.0675646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.0676087Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.0676653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.0677096Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.0677605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.0678100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.0678779Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.0679443Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.0679946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.0680415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.0680752Z dist init r=1, world=2 2023-01-11T22:39:19.0681006Z dist init r=0, world=2 2023-01-11T22:39:19.0681246Z ok (6.114s) 2023-01-11T22:39:19.0681588Z test_input_grads_with_param_mixed_precision (__main__.TestFSDPMixedPrecisionSharded) 2023-01-11T22:39:19.0682167Z Tests that input tensors that require gradients do get their gradients ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74614 2023-01-11T22:39:19.0682711Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74615 2023-01-11T22:39:19.0683320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.0683802Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.0684380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.0684845Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.0685417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.0685841Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.0686410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.0686868Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.0687301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.0687799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.0688454Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.0689136Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.0689633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.0690104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.0690920Z dist init r=0, world=2 2023-01-11T22:39:19.0691158Z dist init r=1, world=2 2023-01-11T22:39:19.0691398Z ok (6.014s) 2023-01-11T22:39:19.0691953Z test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74698 2023-01-11T22:39:19.0692668Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74699 2023-01-11T22:39:19.0693288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.0693719Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.0694290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.0694758Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.0695418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.0695859Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.0696418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.0696882Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.0697312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.0697810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.0698459Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.0699145Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.0699648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.0700116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.0701167Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0702412Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0703649Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0704896Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0706129Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0707345Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0708574Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0709790Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0711011Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0712315Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0713536Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0714751Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0716001Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0717220Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0718442Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0719660Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0720880Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0722092Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0722699Z dist init r=1, world=2 2023-01-11T22:39:19.0722932Z dist init r=0, world=2 2023-01-11T22:39:19.0723175Z ok (6.315s) 2023-01-11T22:39:19.0723706Z test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74782 2023-01-11T22:39:19.0724310Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74783 2023-01-11T22:39:19.0724920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.0725371Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.0726000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.0726452Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.0727020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.0727461Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.0728030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.0728477Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.0728927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.0729425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.0730064Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.0731374Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.0731925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.0732513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.0733580Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0734828Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0736056Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0737281Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0738488Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0739719Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0740943Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0742162Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0743433Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0744649Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0745856Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0747077Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0748336Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0749563Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0750784Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0752004Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0753209Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0754415Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0755635Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0756851Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0758112Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0759329Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0759935Z dist init r=0, world=2 2023-01-11T22:39:19.0760168Z dist init r=1, world=2 2023-01-11T22:39:19.0760406Z ok (6.415s) 2023-01-11T22:39:19.0760963Z test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74866 2023-01-11T22:39:19.0761610Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74867 2023-01-11T22:39:19.0762203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.0762710Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.0763288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.0763739Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.0764323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.0764757Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.0765324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.0765766Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.0766217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.0766723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.0767379Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.0768044Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.0768564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.0769031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.0770022Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0771762Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0772993Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0774306Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0775537Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0776748Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0777973Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0779244Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0780471Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0781668Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0782889Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0784104Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0785319Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0786530Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0787743Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0789077Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0790296Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0791509Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0792101Z dist init r=1, world=2 2023-01-11T22:39:19.0792354Z dist init r=0, world=2 2023-01-11T22:39:19.0792593Z ok (6.314s) 2023-01-11T22:39:19.0793103Z test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74950 2023-01-11T22:39:19.0793717Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74951 2023-01-11T22:39:19.0794370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.0794828Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.0795382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.0795852Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.0796428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.0796871Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.0797417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.0797882Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.0798337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.0798816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.0799472Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.0800158Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.0800675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.0801123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.0802108Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0803347Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0804617Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0805909Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0807132Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0808352Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0809590Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0811248Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0812479Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0813711Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0814921Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0816137Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0817356Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0818572Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0819781Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0821077Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0822286Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0823506Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0824766Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0825987Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0827201Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0828428Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0829031Z dist init r=0, world=2 2023-01-11T22:39:19.0829266Z dist init r=1, world=2 2023-01-11T22:39:19.0829503Z ok (6.315s) 2023-01-11T22:39:19.0830057Z test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75034 2023-01-11T22:39:19.0830688Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75035 2023-01-11T22:39:19.0831296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.0831746Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.0832316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.0832771Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.0833343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.0833786Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.0834334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.0834796Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.0835306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.0835801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.0836437Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.0837133Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.0837651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.0838116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.0839317Z /opt/conda/lib/python3.10/site-packages/torch/_tensor.py:932: UserWarning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:319.) 2023-01-11T22:39:19.0840106Z return iter(self.unbind(0)) 2023-01-11T22:39:19.0841273Z /opt/conda/lib/python3.10/site-packages/torch/_tensor.py:932: UserWarning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:319.) 2023-01-11T22:39:19.0842057Z return iter(self.unbind(0)) 2023-01-11T22:39:19.0842323Z dist init r=1, world=2 2023-01-11T22:39:19.0842553Z dist init r=0, world=2 2023-01-11T22:39:19.0842789Z ok (6.416s) 2023-01-11T22:39:19.0843314Z test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75118 2023-01-11T22:39:19.0843922Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75119 2023-01-11T22:39:19.0844526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.0844978Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.0845549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.0846007Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.0846576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.0847020Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.0847571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.0848030Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.0848477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.0848967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.0849603Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.0850632Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.0851231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.0851699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.0852783Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0854033Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0855259Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0856493Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0857770Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0859005Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0860225Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0861436Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0862660Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0863880Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0865101Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0866321Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0867582Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0868803Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0870010Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0871229Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0872466Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0873692Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0874898Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0876103Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0877312Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0878519Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0879125Z dist init r=1, world=2 2023-01-11T22:39:19.0879357Z dist init r=0, world=2 2023-01-11T22:39:19.0879597Z ok (6.317s) 2023-01-11T22:39:19.0880160Z test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75202 2023-01-11T22:39:19.0880791Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75203 2023-01-11T22:39:19.0881397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.0881895Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.0882468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.0882918Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.0883493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.0883936Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.0884506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.0884951Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.0885404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.0885895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.0886530Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.0887219Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.0887739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.0888250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.0889449Z /opt/conda/lib/python3.10/site-packages/torch/_tensor.py:932: UserWarning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:319.) 2023-01-11T22:39:19.0890226Z return iter(self.unbind(0)) 2023-01-11T22:39:19.0891758Z /opt/conda/lib/python3.10/site-packages/torch/_tensor.py:932: UserWarning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:319.) 2023-01-11T22:39:19.0892547Z return iter(self.unbind(0)) 2023-01-11T22:39:19.0892814Z dist init r=0, world=2 2023-01-11T22:39:19.0893045Z dist init r=1, world=2 2023-01-11T22:39:19.0893284Z ok (6.415s) 2023-01-11T22:39:19.0893807Z test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75286 2023-01-11T22:39:19.0894407Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75287 2023-01-11T22:39:19.0895017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.0895464Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.0896030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.0896484Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.0897060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.0897501Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.0898067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.0898513Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.0899052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.0899543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.0900181Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.0900863Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.0901384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.0901852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.0902820Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0904063Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0905405Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0906640Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0907863Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0909086Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0910305Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0911525Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0912723Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0913930Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0915197Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0916411Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0917623Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0918843Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0920092Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0921359Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0922575Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0923773Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0924978Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0926191Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0927413Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0928630Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0929294Z dist init r=1, world=2 2023-01-11T22:39:19.0929529Z dist init r=0, world=2 2023-01-11T22:39:19.0929768Z ok (6.315s) 2023-01-11T22:39:19.0930758Z test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75370 2023-01-11T22:39:19.0931429Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75371 2023-01-11T22:39:19.0932036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.0932485Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.0933056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.0933528Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.0934081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.0934527Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.0935095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.0935618Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.0936079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.0936571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.0937225Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.0937891Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.0938415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.0938886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.0939879Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0941104Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0942330Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0943563Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0944783Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0946074Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0947301Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0948516Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0949739Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0951000Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0952206Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0953415Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0954630Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0955845Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0957055Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0958275Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0959482Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0960748Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0961333Z dist init r=1, world=2 2023-01-11T22:39:19.0961581Z dist init r=0, world=2 2023-01-11T22:39:19.0961820Z ok (6.315s) 2023-01-11T22:39:19.0962311Z test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75454 2023-01-11T22:39:19.0962917Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75455 2023-01-11T22:39:19.0963526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.0963974Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.0964530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.0964994Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.0965566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.0966007Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.0966606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.0967076Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.0967523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.0967996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.0968654Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.0969343Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.0969861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.0970558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.0971626Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0972857Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0974091Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0975307Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0976532Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0977840Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0979045Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0980262Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0981486Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0982759Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0983980Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0985198Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0986410Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0987623Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0988838Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0990042Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0991250Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0992505Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0993718Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0994924Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0996144Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0997395Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.0998006Z dist init r=0, world=2 2023-01-11T22:39:19.0998240Z dist init r=1, world=2 2023-01-11T22:39:19.0998481Z ok (6.215s) 2023-01-11T22:39:19.0999020Z test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75538 2023-01-11T22:39:19.0999637Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75539 2023-01-11T22:39:19.1000243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1000700Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1001269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1001720Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1002296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1002739Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1003293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1003755Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1004205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.1004741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.1005379Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1006063Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1006573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.1007043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.1008068Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1009303Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1010887Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1012133Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1013436Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1014665Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1015890Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1017088Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1018317Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1019539Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1020757Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1021964Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1023246Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1024458Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1025671Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1026881Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1028121Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1029343Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1029944Z dist init r=0, world=2 2023-01-11T22:39:19.1030195Z dist init r=1, world=2 2023-01-11T22:39:19.1030416Z ok (6.315s) 2023-01-11T22:39:19.1030924Z test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75622 2023-01-11T22:39:19.1031524Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75623 2023-01-11T22:39:19.1032134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1032566Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1033137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1033604Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1034186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1034610Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1035175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1035639Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1036077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.1036569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.1037222Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1037906Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1038473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.1038945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.1039937Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1041168Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1042394Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1043667Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1044886Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1046099Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1047318Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1048534Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1049746Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1051341Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1052564Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1053779Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1055064Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1056280Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1057491Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1058761Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1059983Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1061191Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1062412Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1063619Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1064838Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1066039Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1066645Z dist init r=0, world=2 2023-01-11T22:39:19.1066896Z dist init r=1, world=2 2023-01-11T22:39:19.1067134Z ok (6.214s) 2023-01-11T22:39:19.1067650Z test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75706 2023-01-11T22:39:19.1068281Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75707 2023-01-11T22:39:19.1068942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1069389Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1069942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1070410Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1070986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1071409Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1071977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1072434Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1072887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.1073362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.1074015Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1074744Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1075270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.1075720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.1076932Z /opt/conda/lib/python3.10/site-packages/torch/_tensor.py:932: UserWarning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:319.) 2023-01-11T22:39:19.1077720Z return iter(self.unbind(0)) 2023-01-11T22:39:19.1078838Z /opt/conda/lib/python3.10/site-packages/torch/_tensor.py:932: UserWarning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:319.) 2023-01-11T22:39:19.1079619Z return iter(self.unbind(0)) 2023-01-11T22:39:19.1079865Z dist init r=1, world=2 2023-01-11T22:39:19.1080114Z dist init r=0, world=2 2023-01-11T22:39:19.1080353Z ok (6.415s) 2023-01-11T22:39:19.1080839Z test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75790 2023-01-11T22:39:19.1081445Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75791 2023-01-11T22:39:19.1082053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1082503Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1083060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1083531Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1084108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1084530Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1085093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1085607Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1086053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.1086532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.1087189Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1088336Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1088852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.1089299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.1090547Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1091948Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1093188Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1094403Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1095629Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1096825Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1098055Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1099281Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1100495Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1101781Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1102997Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1104206Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1105459Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1106717Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1107921Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1109122Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1110334Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1111543Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1112745Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1113961Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1115160Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1116369Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1117004Z dist init r=1, world=2 2023-01-11T22:39:19.1117253Z dist init r=0, world=2 2023-01-11T22:39:19.1117490Z ok (6.315s) 2023-01-11T22:39:19.1118010Z test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75874 2023-01-11T22:39:19.1118637Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75875 2023-01-11T22:39:19.1119245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1119691Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1120259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1120716Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1121286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1121722Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1122322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1122787Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1123229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.1123704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.1124356Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1125037Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1125552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.1125999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.1127207Z /opt/conda/lib/python3.10/site-packages/torch/_tensor.py:932: UserWarning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:319.) 2023-01-11T22:39:19.1127994Z return iter(self.unbind(0)) 2023-01-11T22:39:19.1129113Z /opt/conda/lib/python3.10/site-packages/torch/_tensor.py:932: UserWarning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:319.) 2023-01-11T22:39:19.1129896Z return iter(self.unbind(0)) 2023-01-11T22:39:19.1130143Z dist init r=0, world=2 2023-01-11T22:39:19.1130707Z dist init r=1, world=2 2023-01-11T22:39:19.1131059Z ok (6.515s) 2023-01-11T22:39:19.1131530Z test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75958 2023-01-11T22:39:19.1132128Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75959 2023-01-11T22:39:19.1132846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1133292Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1133840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1134305Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1134879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1135316Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1135861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1136316Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1136762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.1137237Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.1137886Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1138563Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1139144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.1139601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.1140585Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1141826Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1143056Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1144274Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1145496Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1146705Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1147906Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1149181Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1150402Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1151616Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1152829Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1154082Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1155301Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1156513Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1157723Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1158913Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1160120Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1161320Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1162520Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1163778Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1164990Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1166185Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1166790Z dist init r=0, world=2 2023-01-11T22:39:19.1167019Z dist init r=1, world=2 2023-01-11T22:39:19.1167255Z ok (6.315s) 2023-01-11T22:39:19.1167787Z test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76042 2023-01-11T22:39:19.1168443Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76043 2023-01-11T22:39:19.1169059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1169504Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1170073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1170710Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1171293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1171735Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1172285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1172746Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1173191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.1173673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.1174304Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1174983Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1175501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.1175965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.1176934Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1178172Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1179396Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1180699Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1181932Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1183170Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1184424Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1185632Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1186848Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1188072Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1189285Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1190491Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1191710Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1192916Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1194124Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1195388Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1196583Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1197787Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1199046Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1200264Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1200861Z dist init r=1, world=2 2023-01-11T22:39:19.1201110Z dist init r=0, world=2 2023-01-11T22:39:19.1201335Z ok (6.315s) 2023-01-11T22:39:19.1201844Z test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76126 2023-01-11T22:39:19.1202447Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76127 2023-01-11T22:39:19.1203038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1203483Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1204050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1204512Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1205106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1205556Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1206116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1206577Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1207003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.1207494Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.1208142Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1208804Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1209320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.1209846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.1210990Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1212240Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1213441Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1214669Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1215964Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1217199Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1218410Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1219631Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1220875Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1222099Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1223309Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1224510Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1225796Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1227018Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1228229Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1229444Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1230695Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1231915Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1233134Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1234329Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1234933Z dist init r=1, world=2 2023-01-11T22:39:19.1235181Z dist init r=0, world=2 2023-01-11T22:39:19.1235414Z ok (6.314s) 2023-01-11T22:39:19.1235930Z test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76210 2023-01-11T22:39:19.1236564Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76211 2023-01-11T22:39:19.1237168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1237616Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1238173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1238639Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1239213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1239633Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1240194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1240709Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1241156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.1241629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.1242282Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1242965Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1243481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.1243927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.1244906Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1246195Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1247414Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1248641Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1249853Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1251234Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1252466Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1253683Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1254891Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1256189Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1257402Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1258619Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1259829Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1261094Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1262303Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1263519Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1264738Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1265938Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1267145Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1268354Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1268957Z dist init r=0, world=2 2023-01-11T22:39:19.1269187Z dist init r=1, world=2 2023-01-11T22:39:19.1269421Z ok (6.216s) 2023-01-11T22:39:19.1269924Z test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76294 2023-01-11T22:39:19.1270561Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76295 2023-01-11T22:39:19.1271171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1271619Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1272185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1272637Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1273204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1273642Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1274204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1274646Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1275096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.1275583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.1276213Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1276942Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1277470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.1277934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.1278897Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1280128Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1281352Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1282575Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1283788Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1285008Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1286217Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1287489Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1288678Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1289890Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1291328Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1292612Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1293835Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1295049Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1296271Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1297475Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1298691Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1299885Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1301100Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1302383Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1302990Z dist init r=1, world=2 2023-01-11T22:39:19.1303239Z dist init r=0, world=2 2023-01-11T22:39:19.1303459Z ok (6.214s) 2023-01-11T22:39:19.1303991Z test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76378 2023-01-11T22:39:19.1304616Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76379 2023-01-11T22:39:19.1305261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1305695Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1306269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1306738Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1307342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1307790Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1308355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1308817Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1309246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.1309738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.1310386Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1311066Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1311569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.1312035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.1313017Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1314245Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1315484Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1316683Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1317967Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1319190Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1320402Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1321620Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1322872Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1324095Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1325306Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1326501Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1327706Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1328914Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1330126Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1331555Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1332853Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1334061Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1335273Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1336482Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1337071Z dist init r=0, world=2 2023-01-11T22:39:19.1337319Z dist init r=1, world=2 2023-01-11T22:39:19.1337553Z ok (6.414s) 2023-01-11T22:39:19.1338095Z test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76462 2023-01-11T22:39:19.1338705Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76463 2023-01-11T22:39:19.1339313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1339758Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1340317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1340778Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1341347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1341790Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1342337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1342794Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1343239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.1343710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.1344363Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1345047Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1345559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.1346012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.1346990Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1348217Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1349501Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1350723Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1351943Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1353202Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1354417Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1355634Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1356855Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1358063Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1359268Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1360478Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1361692Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1362901Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1364174Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1365367Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1366572Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1367827Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1369037Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1370244Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1371066Z dist init r=1, world=2 2023-01-11T22:39:19.1371302Z dist init r=0, world=2 2023-01-11T22:39:19.1371537Z ok (6.214s) 2023-01-11T22:39:19.1372078Z test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76546 2023-01-11T22:39:19.1372709Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76547 2023-01-11T22:39:19.1373304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1373752Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1374321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1374773Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1375343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1375777Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1376341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1376784Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1377228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.1377710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.1378359Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1379108Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1379627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.1380095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.1381085Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1382308Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1383528Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1384796Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1386023Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1387230Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1388450Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1389655Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1390878Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1392080Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1393298Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1394638Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1395859Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1397064Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1398274Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1399530Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1400748Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1401963Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1403155Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1404359Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1405002Z dist init r=0, world=2 2023-01-11T22:39:19.1405251Z dist init r=1, world=2 2023-01-11T22:39:19.1405470Z ok (6.314s) 2023-01-11T22:39:19.1405972Z test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76630 2023-01-11T22:39:19.1406575Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76631 2023-01-11T22:39:19.1407182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1407613Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1408181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1408645Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1409278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1409702Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1410265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1410929Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1411363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.1411853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.1412502Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1413183Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1413412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.1413620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.1414431Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1415185Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1415910Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1416640Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1417358Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1418076Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1418797Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1419507Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1420216Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1420997Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1421713Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1422423Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1423177Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1423890Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1424601Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1425315Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1426021Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1426728Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1427438Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1428144Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1428255Z dist init r=1, world=2 2023-01-11T22:39:19.1428360Z dist init r=0, world=2 2023-01-11T22:39:19.1428506Z ok (6.214s) 2023-01-11T22:39:19.1428934Z test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76714 2023-01-11T22:39:19.1429148Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76715 2023-01-11T22:39:19.1429502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1429674Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1430050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1430237Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1430598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1430773Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1431145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1431333Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1431570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.1431829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.1432232Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1432621Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1432844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.1433069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.1433805Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1434542Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1435263Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1435985Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1436703Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1437429Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1438203Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1438922Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1439642Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1440356Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1441110Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1441823Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1442538Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1443250Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1443956Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1444683Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1445406Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1446121Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1446290Z dist init r=1, world=2 2023-01-11T22:39:19.1446398Z dist init r=0, world=2 2023-01-11T22:39:19.1446481Z ok (6.314s) 2023-01-11T22:39:19.1446881Z test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76798 2023-01-11T22:39:19.1447099Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76799 2023-01-11T22:39:19.1447471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1447641Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1448018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1448208Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1448569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1448722Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1449094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1449322Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1449568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.1449802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.1450196Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1450792Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1451023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.1451245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.1451984Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1452699Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1453429Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1454153Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1454871Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1455664Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1456379Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1457095Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1457811Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1458576Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1459301Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1460015Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1460744Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1461470Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1462187Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1462913Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1463623Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1464392Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1465107Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1465821Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1466533Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1467284Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1467400Z dist init r=0, world=2 2023-01-11T22:39:19.1467507Z dist init r=1, world=2 2023-01-11T22:39:19.1467607Z ok (6.315s) 2023-01-11T22:39:19.1468033Z test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76882 2023-01-11T22:39:19.1468254Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76883 2023-01-11T22:39:19.1468625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1468780Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1469160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1469347Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1469713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1469882Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1470256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1470442Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1470680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.1470912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.1471288Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1471680Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1471903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.1472124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.1472858Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1473640Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1474359Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1475074Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1475827Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1476544Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1477254Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1477970Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1478679Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1479383Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1480114Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1480840Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1481561Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1482330Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1483047Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1483758Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1484512Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1485231Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1485342Z dist init r=0, world=2 2023-01-11T22:39:19.1485449Z dist init r=1, world=2 2023-01-11T22:39:19.1485553Z ok (6.415s) 2023-01-11T22:39:19.1485933Z test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76966 2023-01-11T22:39:19.1486146Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76967 2023-01-11T22:39:19.1486515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1486689Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1487063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1487250Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1487611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1487783Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1488138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1488323Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1488562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.1488796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.1489188Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1489578Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1489799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.1490070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.1490960Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1491700Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1492419Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1493137Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1493920Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1494646Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1495361Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1496074Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1496783Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1497514Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1498245Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1498962Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1499746Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1500464Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1501178Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1501874Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1502628Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1503350Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1504062Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1504820Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1505546Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1506264Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1506378Z dist init r=0, world=2 2023-01-11T22:39:19.1506484Z dist init r=1, world=2 2023-01-11T22:39:19.1506584Z ok (6.314s) 2023-01-11T22:39:19.1507011Z test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77050 2023-01-11T22:39:19.1507226Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77051 2023-01-11T22:39:19.1507596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1507833Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1508208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1508381Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1508744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1508913Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1509283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1509468Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1509706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.1509941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.1510334Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1510704Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1510971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.1511201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.1512176Z /opt/conda/lib/python3.10/site-packages/torch/_tensor.py:932: UserWarning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:319.) 2023-01-11T22:39:19.1512307Z return iter(self.unbind(0)) 2023-01-11T22:39:19.1513274Z /opt/conda/lib/python3.10/site-packages/torch/_tensor.py:932: UserWarning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:319.) 2023-01-11T22:39:19.1513398Z return iter(self.unbind(0)) 2023-01-11T22:39:19.1513507Z dist init r=0, world=2 2023-01-11T22:39:19.1513612Z dist init r=1, world=2 2023-01-11T22:39:19.1513710Z ok (6.414s) 2023-01-11T22:39:19.1514107Z test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77134 2023-01-11T22:39:19.1514310Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77135 2023-01-11T22:39:19.1514677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1514847Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1515219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1515408Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1515768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1515938Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1516310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1516528Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1516767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.1516996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.1517387Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1517777Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1518001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.1518221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.1518959Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1519743Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1520486Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1521247Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1521975Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1522690Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1523403Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1524120Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1524830Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1525536Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1526305Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1527010Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1527717Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1528429Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1529179Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1529896Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1530811Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1531535Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1532246Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1532958Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1533666Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1534375Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1534545Z dist init r=0, world=2 2023-01-11T22:39:19.1534656Z dist init r=1, world=2 2023-01-11T22:39:19.1534754Z ok (6.214s) 2023-01-11T22:39:19.1535183Z test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77218 2023-01-11T22:39:19.1535401Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77219 2023-01-11T22:39:19.1535772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1535943Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1536317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1536508Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1536855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1537024Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1537507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1537702Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1537940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.1538171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.1538565Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1538958Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1539165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.1539385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.1540355Z /opt/conda/lib/python3.10/site-packages/torch/_tensor.py:932: UserWarning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:319.) 2023-01-11T22:39:19.1540483Z return iter(self.unbind(0)) 2023-01-11T22:39:19.1541432Z /opt/conda/lib/python3.10/site-packages/torch/_tensor.py:932: UserWarning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:319.) 2023-01-11T22:39:19.1541558Z return iter(self.unbind(0)) 2023-01-11T22:39:19.1541667Z dist init r=0, world=2 2023-01-11T22:39:19.1541777Z dist init r=1, world=2 2023-01-11T22:39:19.1541875Z ok (6.414s) 2023-01-11T22:39:19.1542267Z test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77302 2023-01-11T22:39:19.1542480Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77303 2023-01-11T22:39:19.1542830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1543050Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1543426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1543612Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1543973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1544143Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1544513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1544700Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1544921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.1545156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.1545545Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1545932Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1546195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.1546419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.1547162Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1547895Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1548621Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1549337Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1550054Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1550768Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1551479Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1552188Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1552951Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1553681Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1554414Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1555177Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1555904Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1556618Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1557333Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1558041Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1558752Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1559465Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1560174Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1560893Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1561665Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1562381Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1562493Z dist init r=1, world=2 2023-01-11T22:39:19.1562588Z dist init r=0, world=2 2023-01-11T22:39:19.1562687Z ok (6.515s) 2023-01-11T22:39:19.1563105Z test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77386 2023-01-11T22:39:19.1563318Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77387 2023-01-11T22:39:19.1563726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1563903Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1564282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1564467Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1564829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1564988Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1565361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1565546Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1565788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.1566020Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.1566411Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1566802Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1567023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.1567232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.1567963Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1568696Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1569425Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1570211Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1571142Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1571859Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1572574Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1573361Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1574082Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1574798Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1575509Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1576216Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1576926Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1577635Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1578340Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1579115Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1579824Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1580530Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1581237Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1581988Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1582106Z dist init r=1, world=2 2023-01-11T22:39:19.1582215Z dist init r=0, world=2 2023-01-11T22:39:19.1582313Z ok (6.214s) 2023-01-11T22:39:19.1582703Z test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77470 2023-01-11T22:39:19.1582922Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77471 2023-01-11T22:39:19.1583289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1583449Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1583825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1584011Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1584374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1584543Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1584919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1585105Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1585345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.1585559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.1585954Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1586343Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1586566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.1586785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.1587577Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1588313Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1589034Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1589757Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1590521Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1591253Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1591976Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1592697Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1593412Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1594124Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1594834Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1595543Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1596301Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1597017Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1597721Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1598433Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1599181Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1599890Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1600603Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1601311Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1601423Z dist init r=0, world=2 2023-01-11T22:39:19.1601531Z dist init r=1, world=2 2023-01-11T22:39:19.1601614Z ok (6.317s) 2023-01-11T22:39:19.1602033Z test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77554 2023-01-11T22:39:19.1602250Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77555 2023-01-11T22:39:19.1602617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1602789Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1603166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1603353Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1603716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1603885Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1604244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1604481Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1604722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.1604997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.1605399Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1605785Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1606011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.1606232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.1606960Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1607742Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1608472Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1609188Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1609908Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1610820Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1611527Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1612244Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1612953Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1613660Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1614454Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1615167Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1615875Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1616641Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1617362Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1618068Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1618786Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1619493Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1620219Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1620950Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1621063Z dist init r=1, world=2 2023-01-11T22:39:19.1621170Z dist init r=0, world=2 2023-01-11T22:39:19.1621268Z ok (6.217s) 2023-01-11T22:39:19.1621658Z test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77638 2023-01-11T22:39:19.1621874Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77639 2023-01-11T22:39:19.1622291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1622461Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1622819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1623006Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1623370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1623539Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1623911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1624097Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1624339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.1624569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.1624960Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1625378Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1625608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.1625831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.1626568Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1627302Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1628029Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1628748Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1629465Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1630180Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1630889Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1631663Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1632374Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1633085Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1633798Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1634553Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1635288Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1636006Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1636725Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1637439Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1638156Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1638867Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1639576Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1640332Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1640443Z dist init r=1, world=2 2023-01-11T22:39:19.1640550Z dist init r=0, world=2 2023-01-11T22:39:19.1640652Z ok (6.317s) 2023-01-11T22:39:19.1641059Z test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77722 2023-01-11T22:39:19.1641274Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77723 2023-01-11T22:39:19.1641643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1641815Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1642189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1642375Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1642776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1642951Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1643324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1643492Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1643732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.1643968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.1644359Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1644749Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1644975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.1645196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.1645929Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1646661Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1647378Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1648093Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1648858Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1649569Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1650484Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1651231Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1652033Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1652762Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1653480Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1654200Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1654911Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1655622Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1656337Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1657057Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1657779Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1658571Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1659290Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1659986Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1660102Z dist init r=0, world=2 2023-01-11T22:39:19.1660209Z dist init r=1, world=2 2023-01-11T22:39:19.1660308Z ok (6.415s) 2023-01-11T22:39:19.1660737Z test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77806 2023-01-11T22:39:19.1660960Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77807 2023-01-11T22:39:19.1661330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1661505Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1661885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1662057Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1662418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1662586Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1662962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1663148Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1663385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.1663617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.1664010Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1664404Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1664611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.1664834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.1665563Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1666290Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1667066Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1667784Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1668497Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1669211Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1669959Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1670677Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1671387Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1672097Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1672802Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1673513Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1674221Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1674929Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1675729Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1676437Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1677144Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1677852Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1678618Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1679352Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1679468Z dist init r=1, world=2 2023-01-11T22:39:19.1679575Z dist init r=0, world=2 2023-01-11T22:39:19.1679673Z ok (6.315s) 2023-01-11T22:39:19.1680092Z test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77890 2023-01-11T22:39:19.1680293Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77891 2023-01-11T22:39:19.1680664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1680834Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1681212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1681402Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1681761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1681930Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1682302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1682489Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1682712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.1682945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.1683339Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1683783Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1684007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.1684230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.1684966Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1685697Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1686418Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1687180Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1687902Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1688619Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1689331Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1690042Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1690959Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1691679Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1692392Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1693190Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1693898Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1694603Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1695313Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1696073Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1696789Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1697497Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1698203Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1698905Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1699020Z dist init r=0, world=2 2023-01-11T22:39:19.1699114Z dist init r=1, world=2 2023-01-11T22:39:19.1699212Z ok (6.414s) 2023-01-11T22:39:19.1699597Z test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77974 2023-01-11T22:39:19.1699812Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77975 2023-01-11T22:39:19.1700181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1700350Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1700723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1700911Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1701253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1701474Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1701849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1702032Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1702273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.1702510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.1702901Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1703288Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1703514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.1703719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.1704491Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1705267Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1705988Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1706712Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1707425Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1708139Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1708854Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1709563Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1710273Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1711057Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1711786Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1712504Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1713262Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1713984Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1714696Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1715423Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1716140Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1716853Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1717568Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1718279Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1718392Z dist init r=1, world=2 2023-01-11T22:39:19.1718500Z dist init r=0, world=2 2023-01-11T22:39:19.1718599Z ok (6.314s) 2023-01-11T22:39:19.1719002Z test_mixed_precision_no_reshard_after_forward (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78058 2023-01-11T22:39:19.1719217Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78059 2023-01-11T22:39:19.1719568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1719743Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1720116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1720303Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1720660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1720830Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1721207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1721391Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1721630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.1721845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.1722282Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1722678Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1722901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.1723122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.1723235Z dist init r=0, world=2 2023-01-11T22:39:19.1723341Z dist init r=1, world=2 2023-01-11T22:39:19.1723439Z ok (6.014s) 2023-01-11T22:39:19.1723635Z test_mixed_precision_resnet (__main__.TestFSDPMixedPrecisionSharded) 2023-01-11T22:39:19.1723852Z End to end test to ensure mixed precision + auto_wrap works ... skip: no torchvision (0.001s) 2023-01-11T22:39:19.1724198Z test_mp_batchnorm_convert_sync_bn_False (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78142 2023-01-11T22:39:19.1724413Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78143 2023-01-11T22:39:19.1724782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1724953Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1725324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1725514Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1725871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1726024Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1726398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1726582Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1726818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.1727048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.1727443Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1727885Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1728107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.1728317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.1728427Z dist init r=0, world=2 2023-01-11T22:39:19.1728537Z dist init r=1, world=2 2023-01-11T22:39:19.1728634Z ok (6.216s) 2023-01-11T22:39:19.1728978Z test_mp_batchnorm_convert_sync_bn_True (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78226 2023-01-11T22:39:19.1729190Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78227 2023-01-11T22:39:19.1729561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1729735Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1730090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1730275Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1730863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1731102Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1731485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1731670Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1731906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.1732138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.1732536Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1732908Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1733132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.1733357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.1733466Z dist init r=1, world=2 2023-01-11T22:39:19.1733572Z dist init r=0, world=2 2023-01-11T22:39:19.1733669Z ok (5.915s) 2023-01-11T22:39:19.1734000Z test_mp_embedding_default (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78310 2023-01-11T22:39:19.1734211Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78311 2023-01-11T22:39:19.1734567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1734739Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1735106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1735296Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1735650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1735817Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1736187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1736372Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1736678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.1736901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.1737297Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1737695Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1737917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.1738137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.1738247Z dist init r=1, world=2 2023-01-11T22:39:19.1738355Z dist init r=0, world=2 2023-01-11T22:39:19.1738451Z ok (6.014s) 2023-01-11T22:39:19.1738782Z test_mp_embedding_only_params_and_bufs (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78394 2023-01-11T22:39:19.1738999Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78395 2023-01-11T22:39:19.1739367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1739536Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1739955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1740144Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1740510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1740680Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1741033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1741223Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1741461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.1741692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.1742088Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1742477Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1742698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.1742918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.1743030Z dist init r=0, world=2 2023-01-11T22:39:19.1743122Z dist init r=1, world=2 2023-01-11T22:39:19.1743220Z ok (6.014s) 2023-01-11T22:39:19.1743569Z test_mp_embedding_params_and_reduce_diff (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78478 2023-01-11T22:39:19.1743782Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78479 2023-01-11T22:39:19.1744148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1744318Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1744687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1744874Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1745217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1745451Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1745824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1746007Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1746245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.1746481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.1746871Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1747256Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1747477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.1747685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.1747794Z dist init r=1, world=2 2023-01-11T22:39:19.1747901Z dist init r=0, world=2 2023-01-11T22:39:19.1748000Z ok (6.114s) 2023-01-11T22:39:19.1748330Z test_mp_embedding_reduce (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78562 2023-01-11T22:39:19.1748590Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78563 2023-01-11T22:39:19.1748964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1749134Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1749485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1749670Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1750033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1750200Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1750572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1750760Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1750998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2023-01-11T22:39:19.1751227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.1751603Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1751988Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-01-11T22:39:19.1752216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.1752436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T22:39:19.1752544Z dist init r=1, world=2 2023-01-11T22:39:19.1752651Z dist init r=0, world=2 2023-01-11T22:39:19.1752749Z ok (6.415s) 2023-01-11T22:39:19.1753091Z test_grads_reduced_precision (__main__.TestFSDPMixedPrecisionUnsharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78646 2023-01-11T22:39:19.1753452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1753606Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1753975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1754225Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1754462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.1754854Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2023-01-11T22:39:19.1755075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.1755601Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:288: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1. 2023-01-11T22:39:19.1755712Z warnings.warn( 2023-01-11T22:39:19.1756451Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1757189Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2023-01-11T22:39:19.1757285Z dist init r=0, world=1 2023-01-11T22:39:19.1757428Z ok (4.710s) 2023-01-11T22:39:19.1757779Z test_mixed_precision_e2e_full_shard (__main__.TestFSDPMixedPrecisionUnsharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78688 2023-01-11T22:39:19.1758146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1758315Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1758691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1758881Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1759118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.1759510Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2023-01-11T22:39:19.1759721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.1760246Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:288: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1. 2023-01-11T22:39:19.1760357Z warnings.warn( 2023-01-11T22:39:19.1760465Z dist init r=0, world=1 2023-01-11T22:39:19.1760564Z ok (4.610s) 2023-01-11T22:39:19.1760922Z test_mixed_precision_no_reshard_after_forward (__main__.TestFSDPMixedPrecisionUnsharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78730 2023-01-11T22:39:19.1761291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T22:39:19.1761461Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T22:39:19.1761818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T22:39:19.1762005Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T22:39:19.1762240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2023-01-11T22:39:19.1762630Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2023-01-11T22:39:19.1762850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T22:39:19.1763433Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:288: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.SHARD_GRAD_OP since the world size is 1. 2023-01-11T22:39:19.1763543Z warnings.warn( 2023-01-11T22:39:19.1763651Z dist init r=0, world=1 2023-01-11T22:39:19.1763734Z ok (4.510s) 2023-01-11T22:39:19.1763772Z 2023-01-11T22:39:19.1764022Z ---------------------------------------------------------------------- 2023-01-11T22:39:19.1764141Z Ran 60 tests in 361.870s 2023-01-11T22:39:19.1764160Z 2023-01-11T22:39:19.1764265Z OK (skipped=1) 2023-01-11T22:39:19.1764284Z 2023-01-11T22:39:19.1764405Z Generating XML reports... 2023-01-11T22:39:19.1764935Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPDifferentSubmodulePrecision-20230111223316.xml 2023-01-11T22:39:19.1765465Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionIgnoredModules-20230111223316.xml 2023-01-11T22:39:19.1765963Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionSharded-20230111223316.xml 2023-01-11T22:39:19.1766458Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionUnsharded-20230111223316.xml 2023-01-11T22:39:19.1766478Z 2023-01-11T22:39:19.1766922Z ##[endgroup] 2023-01-11T22:39:19.1767476Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_mixed_precision (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_mixed_precision_r7m2hb6r) 2023-01-11T22:39:19.1767497Z 2023-01-11T22:39:19.1767787Z Running distributed/rpc/cuda/test_tensorpipe_agent ... [2023-01-11 22:39:19.059490] 2023-01-11T22:39:19.1768309Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/rpc/cuda/test_tensorpipe_agent.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:39:19.059776] 2023-01-11T23:00:11.0870349Z 2023-01-11T23:00:11.0871213Z Expand the folded group to see the log file of distributed/rpc/cuda/test_tensorpipe_agent 2023-01-11T23:00:11.0872809Z ##[group]PRINTING LOG FILE of distributed/rpc/cuda/test_tensorpipe_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-cuda-test_tensorpipe_agent_udxpg47p) 2023-01-11T23:00:11.0873476Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp1g_jsxv 2023-01-11T23:00:11.0874349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp1g_jsxv/_remote_module_non_scriptable.py 2023-01-11T23:00:11.0875363Z ]> 2023-01-11T23:00:11.0876303Z test_ddp_dist_autograd_local_vs_remote_gpu (__main__.TensorPipeCudaDdpComparisonTest) 2023-01-11T23:00:11.0877687Z , <__main__.TensorPipeCudaDistAutogradTest testMethod=test_gpu_to_cpu_continuation>, <__main__.TensorPipeCudaDistAutogradTest testMethod=test_gpu_to_cpu_continuation_gpu_root>]> 2023-01-11T23:00:11.0878889Z test_gpu_simple (__main__.TensorPipeCudaDistAutogradTest) 2023-01-11T23:00:11.0879462Z test_gpu_to_cpu_continuation (__main__.TensorPipeCudaDistAutogradTest) 2023-01-11T23:00:11.0880385Z test_gpu_to_cpu_continuation_gpu_root (__main__.TensorPipeCudaDistAutogradTest) 2023-01-11T23:00:11.0881987Z , <__main__.TensorPipeCudaRemoteModuleTest testMethod=test_input_moved_to_cuda_device_script>, <__main__.TensorPipeCudaRemoteModuleTest testMethod=test_invalid_devices>, <__main__.TensorPipeCudaRemoteModuleTest testMethod=test_valid_device>]> 2023-01-11T23:00:11.0883036Z test_input_moved_to_cuda_device (__main__.TensorPipeCudaRemoteModuleTest) 2023-01-11T23:00:11.0884034Z test_input_moved_to_cuda_device_script (__main__.TensorPipeCudaRemoteModuleTest) 2023-01-11T23:00:11.0884481Z test_invalid_devices (__main__.TensorPipeCudaRemoteModuleTest) 2023-01-11T23:00:11.0884950Z test_valid_device (__main__.TensorPipeCudaRemoteModuleTest) 2023-01-11T23:00:11.0885400Z ]> 2023-01-11T23:00:11.0886178Z test_profiler_remote_cuda (__main__.TensorPipeCudaRpcTest) 2023-01-11T23:00:11.0887517Z , <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_except_last>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_never>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_never_find_unused>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_always>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_except_last>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_never>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_never_find_unused>]> 2023-01-11T23:00:11.0889070Z test_basic_gloo_ckpt_always (__main__.TensorPipePipeWithDDPTest) 2023-01-11T23:00:11.0889621Z test_basic_gloo_ckpt_except_last (__main__.TensorPipePipeWithDDPTest) 2023-01-11T23:00:11.0890036Z test_basic_gloo_ckpt_never (__main__.TensorPipePipeWithDDPTest) 2023-01-11T23:00:11.0891007Z test_basic_gloo_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest) 2023-01-11T23:00:11.0891479Z test_basic_nccl_ckpt_always (__main__.TensorPipePipeWithDDPTest) 2023-01-11T23:00:11.0891895Z test_basic_nccl_ckpt_except_last (__main__.TensorPipePipeWithDDPTest) 2023-01-11T23:00:11.0892293Z test_basic_nccl_ckpt_never (__main__.TensorPipePipeWithDDPTest) 2023-01-11T23:00:11.0892712Z test_basic_nccl_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest) 2023-01-11T23:00:11.0913841Z , <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_async_execution_with_cuda_future>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_callback_changes_devices>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_cuda_sparse_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_cuda_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_custom_class_with_cuda_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_list_with_cuda_sparse_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_list_with_cuda_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_as_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_as_int>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_as_str>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_not_cuda>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_modify_tensor_inplace>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_replace_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_value_on_bad_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream_multi>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream_nested>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream_nested_multi>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_cpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_cpu_to_gpu_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_cpu_to_gpu_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_default_to_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_5>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_6>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_7>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_8>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_5>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_6>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_7>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_8>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_non_default_to_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_to_cpu_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_to_cpu_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_gpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_in_options>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_invalid_max_local_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_invalid_max_remote_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_invalid_min_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_many_to_one>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_loop>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_not_timeout>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_remote>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_remote_response>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_response>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_response_loop>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_multi_gpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_multi_gpu_self>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_one_to_many>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_remote>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_return_to_gpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_return_to_gpu_self>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_wrong_worker_name>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_mismatch>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_devices_option_mismatch>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_devices_option_mismatch_reverse>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization5>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_with_unpickleable_attributes>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_tensor_view_as_return_value>]> 2023-01-11T23:00:11.0938940Z test_async_execution_nested_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0939915Z test_async_execution_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0940918Z test_cuda_future_callback_changes_devices (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0941931Z test_cuda_future_can_extract_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0942897Z test_cuda_future_can_extract_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0943773Z test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0944884Z test_cuda_future_can_extract_custom_class_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0945899Z test_cuda_future_can_extract_list_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0946930Z test_cuda_future_can_extract_list_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0947919Z test_cuda_future_device_as_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0948742Z test_cuda_future_device_as_int (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0949541Z test_cuda_future_device_as_str (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0950394Z test_cuda_future_device_not_cuda (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0951342Z test_cuda_future_modify_tensor_inplace (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0953040Z test_cuda_future_replace_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0953555Z test_cuda_future_value_on_bad_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0954028Z test_custom_stream (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0954597Z test_custom_stream_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0955117Z test_custom_stream_nested (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0955603Z test_custom_stream_nested_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0956081Z test_device_map_cpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0956552Z test_device_map_cpu_to_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0957050Z test_device_map_cpu_to_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0957548Z test_device_map_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0958031Z test_device_map_gpu_default_to_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0958530Z test_device_map_gpu_mixed_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0959010Z test_device_map_gpu_mixed_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0959475Z test_device_map_gpu_mixed_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0959929Z test_device_map_gpu_mixed_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0960404Z test_device_map_gpu_mixed_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0960868Z test_device_map_gpu_mixed_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0961394Z test_device_map_gpu_mixed_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0961875Z test_device_map_gpu_mixed_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0962351Z test_device_map_gpu_mixed_self_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0962837Z test_device_map_gpu_mixed_self_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0963305Z test_device_map_gpu_mixed_self_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0963791Z test_device_map_gpu_mixed_self_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0964258Z test_device_map_gpu_mixed_self_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0964731Z test_device_map_gpu_mixed_self_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0965184Z test_device_map_gpu_mixed_self_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0965661Z test_device_map_gpu_mixed_self_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0966136Z test_device_map_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0966622Z test_device_map_gpu_non_default_to_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0967127Z test_device_map_gpu_to_cpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0967631Z test_device_map_gpu_to_cpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0968120Z test_device_maps_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0968568Z test_device_maps_in_options (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0969067Z test_device_maps_invalid_max_local_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0969595Z test_device_maps_invalid_max_remote_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0970097Z test_device_maps_invalid_min_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0971157Z test_device_maps_many_to_one (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0971654Z test_device_maps_missing_config (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0972162Z test_device_maps_missing_config_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0972649Z test_device_maps_missing_config_not_timeout (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0973260Z test_device_maps_missing_config_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0973793Z test_device_maps_missing_config_remote_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0974290Z test_device_maps_missing_config_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0974817Z test_device_maps_missing_config_response_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0975315Z test_device_maps_multi_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0975800Z test_device_maps_multi_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0976259Z test_device_maps_one_to_many (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0976734Z test_device_maps_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0977199Z test_device_maps_return_to_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0977674Z test_device_maps_return_to_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0978175Z test_device_maps_wrong_worker_name (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0978660Z test_device_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0979137Z test_devices_option_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0979612Z test_devices_option_mismatch_reverse (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0980184Z test_owner_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0980716Z test_owner_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0981247Z test_owner_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0981729Z test_owner_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0982243Z test_rref_as_arg_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0982744Z test_rref_as_arg_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0983214Z test_rref_as_arg_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0983706Z test_rref_as_arg_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0984201Z test_rref_as_arg_synchronization5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0984691Z test_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0985176Z test_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0985688Z test_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0986183Z test_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0986654Z test_rref_to_here_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0987157Z test_rref_to_here_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0987664Z test_rref_to_here_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0988144Z test_rref_to_here_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0988627Z test_rref_with_unpickleable_attributes (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0989126Z test_tensor_view_as_return_value (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2023-01-11T23:00:11.0990039Z , <__main__.TensorPipeTensorPipeCudaDistAutogradTest testMethod=test_dist_autograd_sync_streams>, <__main__.TensorPipeTensorPipeCudaDistAutogradTest testMethod=test_gradients_synchronizations>]> 2023-01-11T23:00:11.0990940Z test_device_maps_backward_pass (__main__.TensorPipeTensorPipeCudaDistAutogradTest) 2023-01-11T23:00:11.0991495Z test_dist_autograd_sync_streams (__main__.TensorPipeTensorPipeCudaDistAutogradTest) 2023-01-11T23:00:11.0992019Z test_gradients_synchronizations (__main__.TensorPipeTensorPipeCudaDistAutogradTest) 2023-01-11T23:00:11.0992782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.0993240Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.0993814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.0994274Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.0994749Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk3km5x5v 2023-01-11T23:00:11.0995270Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk3km5x5v/_remote_module_non_scriptable.py 2023-01-11T23:00:11.0995579Z 2023-01-11T23:00:11.0995689Z Running tests... 2023-01-11T23:00:11.0996099Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.0996679Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.0997249Z test_ddp_dist_autograd_local_vs_remote_gpu (__main__.TensorPipeCudaDdpComparisonTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.0997834Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78840 2023-01-11T23:00:11.0998291Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78841 2023-01-11T23:00:11.0998718Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 78842 2023-01-11T23:00:11.0999155Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 78843 2023-01-11T23:00:11.0999760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1000215Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1000769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1001232Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1001803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1002240Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1002789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1003245Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1003814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1004236Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1004796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1005248Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1005817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1006237Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1006806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1007256Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1007693Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqetql25s 2023-01-11T23:00:11.1008223Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqetql25s/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1008810Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqauh87k3 2023-01-11T23:00:11.1009338Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqauh87k3/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1009840Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp20qwad03 2023-01-11T23:00:11.1010971Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp20qwad03/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1011557Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd1rizk10 2023-01-11T23:00:11.1012084Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd1rizk10/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1012568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1013034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.1013499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.1013939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1014328Z skip: Need at least 4 CUDA devices (4.504s) 2023-01-11T23:00:11.1014520Z 2023-01-11T23:00:11.1014803Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1015130Z Ran 1 test in 4.504s 2023-01-11T23:00:11.1015379Z 2023-01-11T23:00:11.1015503Z OK (skipped=1) 2023-01-11T23:00:11.1015658Z 2023-01-11T23:00:11.1015785Z Generating XML reports... 2023-01-11T23:00:11.1016466Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDdpComparisonTest-20230111223923.xml 2023-01-11T23:00:11.1017214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1017664Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1018234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1018699Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1019143Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgjo9chov 2023-01-11T23:00:11.1019680Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgjo9chov/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1019978Z 2023-01-11T23:00:11.1020088Z Running tests... 2023-01-11T23:00:11.1020491Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1021041Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1021581Z test_gpu_simple (__main__.TensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1022073Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79011 2023-01-11T23:00:11.1022496Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79012 2023-01-11T23:00:11.1022927Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 79013 2023-01-11T23:00:11.1023368Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 79014 2023-01-11T23:00:11.1023971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1024402Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1024975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1025442Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1025998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1026521Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1027090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1027555Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1028111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1028552Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1029124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1029564Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1030138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1030581Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1031145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1031584Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1032043Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp01kgf_vo 2023-01-11T23:00:11.1032629Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp01kgf_vo/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1033163Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbbpmmznn 2023-01-11T23:00:11.1033676Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbbpmmznn/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1034204Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkq69y7fc 2023-01-11T23:00:11.1034735Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkq69y7fc/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1035244Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdxzrfidj 2023-01-11T23:00:11.1035772Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdxzrfidj/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1036272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.1036741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.1037184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1037635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1038025Z fi_getinfo: -61 2023-01-11T23:00:11.1038282Z fi_getinfo: -61 2023-01-11T23:00:11.1038553Z fi_getinfo: -61 2023-01-11T23:00:11.1038823Z fi_getinfo: -61 2023-01-11T23:00:11.1039039Z ok (6.669s) 2023-01-11T23:00:11.1039189Z 2023-01-11T23:00:11.1039457Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1039787Z Ran 1 test in 6.669s 2023-01-11T23:00:11.1039951Z 2023-01-11T23:00:11.1040044Z OK 2023-01-11T23:00:11.1040159Z 2023-01-11T23:00:11.1040285Z Generating XML reports... 2023-01-11T23:00:11.1040948Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20230111223930.xml 2023-01-11T23:00:11.1041707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1042138Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1042709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1043169Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1043631Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ytficxp 2023-01-11T23:00:11.1044211Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ytficxp/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1044508Z 2023-01-11T23:00:11.1044618Z Running tests... 2023-01-11T23:00:11.1045024Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1045570Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1046132Z test_gpu_to_cpu_continuation (__main__.TensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1046637Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79434 2023-01-11T23:00:11.1047074Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79435 2023-01-11T23:00:11.1047497Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 79436 2023-01-11T23:00:11.1047929Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 79437 2023-01-11T23:00:11.1048533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1048978Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1049529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1050039Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1050929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1051351Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1051906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1052345Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1052916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1053360Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1053940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1054393Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1054988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1055430Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1055993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1056450Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1056893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp89ceco40 2023-01-11T23:00:11.1057424Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp89ceco40/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1057944Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3_h9l4h3 2023-01-11T23:00:11.1058456Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8f0qvss5 2023-01-11T23:00:11.1058961Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3_h9l4h3/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1059494Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8f0qvss5/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1060013Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjbn9rkbz 2023-01-11T23:00:11.1060520Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjbn9rkbz/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1061018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.1061567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1062029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.1062468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1062847Z fi_getinfo: -61 2023-01-11T23:00:11.1063131Z fi_getinfo: -61 2023-01-11T23:00:11.1063389Z fi_getinfo: -61 2023-01-11T23:00:11.1063659Z fi_getinfo: -61 2023-01-11T23:00:11.1063891Z ok (6.701s) 2023-01-11T23:00:11.1064036Z 2023-01-11T23:00:11.1064287Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1064613Z Ran 1 test in 6.701s 2023-01-11T23:00:11.1064775Z 2023-01-11T23:00:11.1064870Z OK 2023-01-11T23:00:11.1065001Z 2023-01-11T23:00:11.1065107Z Generating XML reports... 2023-01-11T23:00:11.1065781Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20230111223940.xml 2023-01-11T23:00:11.1066546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1066993Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1067610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1068085Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1068547Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqe9h4xs8 2023-01-11T23:00:11.1069075Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqe9h4xs8/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1069351Z 2023-01-11T23:00:11.1069460Z Running tests... 2023-01-11T23:00:11.1069864Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1070437Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1070988Z test_gpu_to_cpu_continuation_gpu_root (__main__.TensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1071503Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79857 2023-01-11T23:00:11.1071949Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79858 2023-01-11T23:00:11.1072386Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 79859 2023-01-11T23:00:11.1072804Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 79860 2023-01-11T23:00:11.1073403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1073847Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1074402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1074866Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1075439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1075876Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1076425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1076885Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1077547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1077978Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1078520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1079042Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1079612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1080029Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1080601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1081057Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1081517Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0jgyymbe 2023-01-11T23:00:11.1082033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0jgyymbe/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1082562Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfzu8lkox 2023-01-11T23:00:11.1083090Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfzu8lkox/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1083593Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppmb2rrpn 2023-01-11T23:00:11.1084120Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppmb2rrpn/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1084647Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4be9438m 2023-01-11T23:00:11.1085217Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4be9438m/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1085708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1086175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.1086637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.1087099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1087469Z fi_getinfo: -61 2023-01-11T23:00:11.1087746Z fi_getinfo: -61 2023-01-11T23:00:11.1088018Z fi_getinfo: -61 2023-01-11T23:00:11.1088273Z fi_getinfo: -61 2023-01-11T23:00:11.1088504Z ok (6.834s) 2023-01-11T23:00:11.1088649Z 2023-01-11T23:00:11.1088915Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1089224Z Ran 1 test in 6.834s 2023-01-11T23:00:11.1089387Z 2023-01-11T23:00:11.1089482Z OK 2023-01-11T23:00:11.1089615Z 2023-01-11T23:00:11.1089739Z Generating XML reports... 2023-01-11T23:00:11.1090714Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20230111223949.xml 2023-01-11T23:00:11.1091490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1091941Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1092516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1092963Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1093424Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8g9uj0n1 2023-01-11T23:00:11.1093961Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8g9uj0n1/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1094259Z 2023-01-11T23:00:11.1094368Z Running tests... 2023-01-11T23:00:11.1094751Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1095319Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1095881Z test_input_moved_to_cuda_device (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1096461Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80280 2023-01-11T23:00:11.1096903Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80281 2023-01-11T23:00:11.1097506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1097953Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1098507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1098972Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1099544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1099961Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1100527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1100989Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1101448Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2qm6hmsf 2023-01-11T23:00:11.1101962Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2qm6hmsf/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1102488Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4yfm3fu2 2023-01-11T23:00:11.1103076Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4yfm3fu2/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1103591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1104037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1104417Z fi_getinfo: -61 2023-01-11T23:00:11.1104693Z fi_getinfo: -61 2023-01-11T23:00:11.1104911Z ok (6.126s) 2023-01-11T23:00:11.1105060Z 2023-01-11T23:00:11.1105324Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1105650Z Ran 1 test in 6.126s 2023-01-11T23:00:11.1105809Z 2023-01-11T23:00:11.1105884Z OK 2023-01-11T23:00:11.1106017Z 2023-01-11T23:00:11.1106142Z Generating XML reports... 2023-01-11T23:00:11.1106812Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20230111223959.xml 2023-01-11T23:00:11.1107572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1107998Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1108568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1109032Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1109477Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw_v11jaz 2023-01-11T23:00:11.1110013Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw_v11jaz/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1110306Z 2023-01-11T23:00:11.1110416Z Running tests... 2023-01-11T23:00:11.1110815Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1111367Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1111937Z test_input_moved_to_cuda_device_script (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1112455Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80470 2023-01-11T23:00:11.1112877Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80471 2023-01-11T23:00:11.1113477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1113982Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1114556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1115002Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1115574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1116014Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1116585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1117025Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1117484Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpusok1ssp 2023-01-11T23:00:11.1118020Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpusok1ssp/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1118535Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphwubw6g3 2023-01-11T23:00:11.1119062Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphwubw6g3/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1119562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1120081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1120458Z fi_getinfo: -61 2023-01-11T23:00:11.1120731Z fi_getinfo: -61 2023-01-11T23:00:11.1121234Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpusok1ssp/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2023-01-11T23:00:11.1121940Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphwubw6g3/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2023-01-11T23:00:11.1122678Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmphwubw6g3/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2023-01-11T23:00:11.1123149Z ok (6.202s) 2023-01-11T23:00:11.1123295Z 2023-01-11T23:00:11.1123565Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1123871Z Ran 1 test in 6.202s 2023-01-11T23:00:11.1124032Z 2023-01-11T23:00:11.1124128Z OK 2023-01-11T23:00:11.1124262Z 2023-01-11T23:00:11.1124385Z Generating XML reports... 2023-01-11T23:00:11.1125052Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20230111224008.xml 2023-01-11T23:00:11.1125795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1126242Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1126820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1127268Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1127727Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwpmc611r 2023-01-11T23:00:11.1128266Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwpmc611r/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1128561Z 2023-01-11T23:00:11.1128671Z Running tests... 2023-01-11T23:00:11.1129055Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1129623Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1130179Z test_invalid_devices (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1131041Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80676 2023-01-11T23:00:11.1131581Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80677 2023-01-11T23:00:11.1132191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1132639Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1133194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1133659Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1134231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1134667Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1135213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1135668Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1136122Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_dxnfid7 2023-01-11T23:00:11.1136639Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_dxnfid7/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1137163Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5bvcmx7v 2023-01-11T23:00:11.1137843Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5bvcmx7v/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1138359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1138808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1139189Z fi_getinfo: -61 2023-01-11T23:00:11.1139468Z fi_getinfo: -61 2023-01-11T23:00:11.1139717Z On WorkerInfo(id=1, name=worker1): 2023-01-11T23:00:11.1140622Z RuntimeError('CUDA error: invalid device ordinal\nCUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.\nCompile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.\n') 2023-01-11T23:00:11.1141272Z Traceback (most recent call last): 2023-01-11T23:00:11.1141786Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function 2023-01-11T23:00:11.1142225Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2023-01-11T23:00:11.1142797Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/nn/api/remote_module.py", line 96, in _create_module 2023-01-11T23:00:11.1143176Z module.to(device) 2023-01-11T23:00:11.1143606Z File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1132, in to 2023-01-11T23:00:11.1143967Z return self._apply(convert) 2023-01-11T23:00:11.1144445Z File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 807, in _apply 2023-01-11T23:00:11.1144811Z param_applied = fn(param) 2023-01-11T23:00:11.1145268Z File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in convert 2023-01-11T23:00:11.1145723Z return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking) 2023-01-11T23:00:11.1146114Z RuntimeError: CUDA error: invalid device ordinal 2023-01-11T23:00:11.1146543Z CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. 2023-01-11T23:00:11.1146989Z For debugging consider passing CUDA_LAUNCH_BLOCKING=1. 2023-01-11T23:00:11.1147442Z Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. 2023-01-11T23:00:11.1147669Z 2023-01-11T23:00:11.1147687Z 2023-01-11T23:00:11.1147820Z On WorkerInfo(id=1, name=worker1): 2023-01-11T23:00:11.1150968Z RuntimeError('On WorkerInfo(id=1, name=worker1):\nRuntimeError(\'CUDA error: invalid device ordinal\nCUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.\nCompile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.\n\')\nTraceback (most recent call last):\n File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function\n result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n File "/opt/conda/lib/python3.10/site-packages/torch/distributed/nn/api/remote_module.py", line 96, in _create_module\n module.to(device)\n File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1132, in to\n return self._apply(convert)\n File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 807, in _apply\n param_applied = fn(param)\n File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in convert\n return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)\nRuntimeError: CUDA error: invalid device ordinal\nCUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.\nCompile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.\n\n') 2023-01-11T23:00:11.1153187Z Traceback (most recent call last): 2023-01-11T23:00:11.1153753Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function 2023-01-11T23:00:11.1154216Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2023-01-11T23:00:11.1154616Z File "/tmp/tmpwpmc611r/_remote_module_non_scriptable.py", line 47, in _remote_forward 2023-01-11T23:00:11.1155034Z module = module_rref.local_value() 2023-01-11T23:00:11.1155569Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 234, in _handle_exception 2023-01-11T23:00:11.1155943Z raise exc 2023-01-11T23:00:11.1156216Z RuntimeError: On WorkerInfo(id=1, name=worker1): 2023-01-11T23:00:11.1156617Z RuntimeError('CUDA error: invalid device ordinal 2023-01-11T23:00:11.1157067Z CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. 2023-01-11T23:00:11.1157498Z For debugging consider passing CUDA_LAUNCH_BLOCKING=1. 2023-01-11T23:00:11.1157950Z Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. 2023-01-11T23:00:11.1158281Z ') 2023-01-11T23:00:11.1158513Z Traceback (most recent call last): 2023-01-11T23:00:11.1159029Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function 2023-01-11T23:00:11.1159476Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2023-01-11T23:00:11.1160050Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/nn/api/remote_module.py", line 96, in _create_module 2023-01-11T23:00:11.1160416Z module.to(device) 2023-01-11T23:00:11.1160870Z File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1132, in to 2023-01-11T23:00:11.1161233Z return self._apply(convert) 2023-01-11T23:00:11.1161689Z File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 807, in _apply 2023-01-11T23:00:11.1162058Z param_applied = fn(param) 2023-01-11T23:00:11.1162541Z File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in convert 2023-01-11T23:00:11.1163008Z return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking) 2023-01-11T23:00:11.1163386Z RuntimeError: CUDA error: invalid device ordinal 2023-01-11T23:00:11.1163835Z CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. 2023-01-11T23:00:11.1164283Z For debugging consider passing CUDA_LAUNCH_BLOCKING=1. 2023-01-11T23:00:11.1164717Z Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. 2023-01-11T23:00:11.1165005Z 2023-01-11T23:00:11.1165023Z 2023-01-11T23:00:11.1165041Z 2023-01-11T23:00:11.1165147Z ok (4.601s) 2023-01-11T23:00:11.1165295Z 2023-01-11T23:00:11.1165562Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1165889Z Ran 1 test in 4.601s 2023-01-11T23:00:11.1166029Z 2023-01-11T23:00:11.1166124Z OK 2023-01-11T23:00:11.1166257Z 2023-01-11T23:00:11.1166383Z Generating XML reports... 2023-01-11T23:00:11.1167063Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20230111224017.xml 2023-01-11T23:00:11.1167806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1168254Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1168824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1169293Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1169735Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5o76n36o 2023-01-11T23:00:11.1170328Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5o76n36o/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1170837Z 2023-01-11T23:00:11.1170954Z Running tests... 2023-01-11T23:00:11.1171496Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1172062Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1172608Z test_valid_device (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1173106Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80865 2023-01-11T23:00:11.1173533Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80866 2023-01-11T23:00:11.1174137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1174584Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1175155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1175606Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1176183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1176622Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1177174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1177646Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1178208Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_rmsyete 2023-01-11T23:00:11.1178740Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_rmsyete/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1179253Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps3bc7pni 2023-01-11T23:00:11.1179788Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps3bc7pni/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1180297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1180768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1181140Z fi_getinfo: -61 2023-01-11T23:00:11.1181419Z fi_getinfo: -61 2023-01-11T23:00:11.1181656Z ok (6.023s) 2023-01-11T23:00:11.1181784Z 2023-01-11T23:00:11.1182051Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1182375Z Ran 1 test in 6.023s 2023-01-11T23:00:11.1182619Z 2023-01-11T23:00:11.1182714Z OK 2023-01-11T23:00:11.1182850Z 2023-01-11T23:00:11.1182957Z Generating XML reports... 2023-01-11T23:00:11.1183631Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20230111224024.xml 2023-01-11T23:00:11.1184394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1184843Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1185396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1185863Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1186327Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq541uo1y 2023-01-11T23:00:11.1186843Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq541uo1y/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1187145Z 2023-01-11T23:00:11.1187256Z Running tests... 2023-01-11T23:00:11.1187657Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1188231Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1188746Z test_profiler_remote_cuda (__main__.TensorPipeCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1189278Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81055 2023-01-11T23:00:11.1189807Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81056 2023-01-11T23:00:11.1190236Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 81057 2023-01-11T23:00:11.1190674Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 81058 2023-01-11T23:00:11.1191281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1191734Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1192278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1192717Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1193290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1193757Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1194318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1194776Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1195345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1195773Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1196338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1196798Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1197371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1197791Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1198358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1198812Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1199252Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphxxhgsyc 2023-01-11T23:00:11.1199786Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphxxhgsyc/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1200374Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcm1_lgmv 2023-01-11T23:00:11.1200901Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcm1_lgmv/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1201405Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1cyqpphs 2023-01-11T23:00:11.1201939Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1cyqpphs/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1202461Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp88ixfnod 2023-01-11T23:00:11.1202984Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp88ixfnod/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1203468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1204048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1204513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.1204957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.1205341Z fi_getinfo: -61 2023-01-11T23:00:11.1205618Z fi_getinfo: -61 2023-01-11T23:00:11.1205871Z fi_getinfo: -61 2023-01-11T23:00:11.1206143Z fi_getinfo: -61 2023-01-11T23:00:11.1206379Z ok (9.097s) 2023-01-11T23:00:11.1206526Z 2023-01-11T23:00:11.1206819Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1207155Z Ran 1 test in 9.098s 2023-01-11T23:00:11.1207315Z 2023-01-11T23:00:11.1207411Z OK 2023-01-11T23:00:11.1207548Z 2023-01-11T23:00:11.1207676Z Generating XML reports... 2023-01-11T23:00:11.1208295Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRpcTest-20230111224033.xml 2023-01-11T23:00:11.1209032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1209478Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1210031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1210821Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1211291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnp0uuwgy 2023-01-11T23:00:11.1211836Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnp0uuwgy/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1212136Z 2023-01-11T23:00:11.1212229Z Running tests... 2023-01-11T23:00:11.1212635Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1213209Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1213761Z test_basic_gloo_ckpt_always (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1214239Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81402 2023-01-11T23:00:11.1214686Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81403 2023-01-11T23:00:11.1215290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1215720Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1216288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1216754Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1217324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1217742Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1218399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1218860Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1219304Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0w8jqrje 2023-01-11T23:00:11.1219844Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0w8jqrje/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1220374Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuo3ggbxx 2023-01-11T23:00:11.1220903Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuo3ggbxx/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1221392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1221861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1222253Z skip: Need at least 4 CUDA devices (4.263s) 2023-01-11T23:00:11.1222445Z 2023-01-11T23:00:11.1222714Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1223022Z Ran 1 test in 4.263s 2023-01-11T23:00:11.1223184Z 2023-01-11T23:00:11.1223294Z OK (skipped=1) 2023-01-11T23:00:11.1223447Z 2023-01-11T23:00:11.1223574Z Generating XML reports... 2023-01-11T23:00:11.1224270Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111224045.xml 2023-01-11T23:00:11.1225030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1225480Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1226053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1226507Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1226969Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjvp04sdn 2023-01-11T23:00:11.1227502Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjvp04sdn/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1227803Z 2023-01-11T23:00:11.1227894Z Running tests... 2023-01-11T23:00:11.1228295Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1228868Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1229422Z test_basic_gloo_ckpt_except_last (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1229905Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81505 2023-01-11T23:00:11.1230351Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81506 2023-01-11T23:00:11.1230958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1231384Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1231958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1232422Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1232997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1233417Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1233991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1234449Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1234909Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp18r9e8y6 2023-01-11T23:00:11.1235483Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp18r9e8y6/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1236018Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplo8wa9a3 2023-01-11T23:00:11.1236547Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplo8wa9a3/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1237038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1237506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1237889Z skip: Need at least 4 CUDA devices (4.304s) 2023-01-11T23:00:11.1238079Z 2023-01-11T23:00:11.1238349Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1238660Z Ran 1 test in 4.304s 2023-01-11T23:00:11.1238821Z 2023-01-11T23:00:11.1238931Z OK (skipped=1) 2023-01-11T23:00:11.1239090Z 2023-01-11T23:00:11.1239215Z Generating XML reports... 2023-01-11T23:00:11.1239844Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111224052.xml 2023-01-11T23:00:11.1240589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1241035Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1241652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1242108Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1242573Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdntwbtv3 2023-01-11T23:00:11.1243113Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdntwbtv3/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1243411Z 2023-01-11T23:00:11.1243527Z Running tests... 2023-01-11T23:00:11.1243914Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1244486Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1245037Z test_basic_gloo_ckpt_never (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1245514Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81608 2023-01-11T23:00:11.1245961Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81609 2023-01-11T23:00:11.1246564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1247007Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1247556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1248022Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1248591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1249009Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1249580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1250043Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1250719Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0wzncmwe 2023-01-11T23:00:11.1251246Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0wzncmwe/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1251773Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfe8_oygo 2023-01-11T23:00:11.1252300Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfe8_oygo/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1252865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1253334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1253715Z skip: Need at least 4 CUDA devices (4.398s) 2023-01-11T23:00:11.1253903Z 2023-01-11T23:00:11.1254183Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1254496Z Ran 1 test in 4.399s 2023-01-11T23:00:11.1254657Z 2023-01-11T23:00:11.1254805Z OK (skipped=1) 2023-01-11T23:00:11.1254961Z 2023-01-11T23:00:11.1255089Z Generating XML reports... 2023-01-11T23:00:11.1255724Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111224059.xml 2023-01-11T23:00:11.1256463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1256910Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1257481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1257927Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1258387Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdf84xcw0 2023-01-11T23:00:11.1259000Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdf84xcw0/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1259307Z 2023-01-11T23:00:11.1259418Z Running tests... 2023-01-11T23:00:11.1259803Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1260376Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1260939Z test_basic_gloo_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1261431Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81711 2023-01-11T23:00:11.1261874Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81712 2023-01-11T23:00:11.1262473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1262920Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1263479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1263949Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1264521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1264941Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1265508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1265972Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1266433Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp56hfg3ta 2023-01-11T23:00:11.1266947Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp56hfg3ta/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1267478Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaybfannh 2023-01-11T23:00:11.1268009Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaybfannh/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1268513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1268962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1269346Z skip: Need at least 4 CUDA devices (4.360s) 2023-01-11T23:00:11.1269594Z 2023-01-11T23:00:11.1269864Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1270170Z Ran 1 test in 4.360s 2023-01-11T23:00:11.1270332Z 2023-01-11T23:00:11.1270443Z OK (skipped=1) 2023-01-11T23:00:11.1270597Z 2023-01-11T23:00:11.1270722Z Generating XML reports... 2023-01-11T23:00:11.1271377Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111224106.xml 2023-01-11T23:00:11.1272109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1272563Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1273136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1273585Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1274046Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp2i0b88w 2023-01-11T23:00:11.1274584Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp2i0b88w/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1274878Z 2023-01-11T23:00:11.1274989Z Running tests... 2023-01-11T23:00:11.1275376Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1275947Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1276539Z test_basic_nccl_ckpt_always (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1277044Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81814 2023-01-11T23:00:11.1277470Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81815 2023-01-11T23:00:11.1278069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1278520Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1279070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1279539Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1280112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1280555Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1281103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1281563Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1282027Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdwp2pkwe 2023-01-11T23:00:11.1282547Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdwp2pkwe/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1283082Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7mxg03ls 2023-01-11T23:00:11.1283610Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7mxg03ls/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1284111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1284563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1284947Z skip: Need at least 4 CUDA devices (4.367s) 2023-01-11T23:00:11.1285136Z 2023-01-11T23:00:11.1285405Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1285714Z Ran 1 test in 4.367s 2023-01-11T23:00:11.1285875Z 2023-01-11T23:00:11.1285985Z OK (skipped=1) 2023-01-11T23:00:11.1286139Z 2023-01-11T23:00:11.1286265Z Generating XML reports... 2023-01-11T23:00:11.1286914Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111224113.xml 2023-01-11T23:00:11.1287699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1288145Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1288715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1289186Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1289629Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpln4vr2pb 2023-01-11T23:00:11.1290162Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpln4vr2pb/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1290678Z 2023-01-11T23:00:11.1290795Z Running tests... 2023-01-11T23:00:11.1291187Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1291767Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1292324Z test_basic_nccl_ckpt_except_last (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1292826Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81917 2023-01-11T23:00:11.1293249Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81918 2023-01-11T23:00:11.1293918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1294376Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1294930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1295397Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1295968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1296411Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1296962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1297429Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1297893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiim66dci 2023-01-11T23:00:11.1298414Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiim66dci/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1298940Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcirzsi45 2023-01-11T23:00:11.1299469Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcirzsi45/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1299972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1300427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1300809Z skip: Need at least 4 CUDA devices (4.310s) 2023-01-11T23:00:11.1300997Z 2023-01-11T23:00:11.1301270Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1301597Z Ran 1 test in 4.310s 2023-01-11T23:00:11.1301738Z 2023-01-11T23:00:11.1301848Z OK (skipped=1) 2023-01-11T23:00:11.1302002Z 2023-01-11T23:00:11.1302130Z Generating XML reports... 2023-01-11T23:00:11.1302777Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111224120.xml 2023-01-11T23:00:11.1303497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1303940Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1304507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1305049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1305494Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpixtzvcb1 2023-01-11T23:00:11.1306034Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpixtzvcb1/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1306329Z 2023-01-11T23:00:11.1306444Z Running tests... 2023-01-11T23:00:11.1306831Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1307401Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1307947Z test_basic_nccl_ckpt_never (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1308437Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82020 2023-01-11T23:00:11.1308871Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82021 2023-01-11T23:00:11.1309467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1309909Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1310458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1310972Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1311557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1312000Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1312552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1313015Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1313482Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplfhprhhz 2023-01-11T23:00:11.1314027Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplfhprhhz/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1314536Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp395gm5sj 2023-01-11T23:00:11.1315067Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp395gm5sj/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1315568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1316015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1316405Z skip: Need at least 4 CUDA devices (4.358s) 2023-01-11T23:00:11.1316594Z 2023-01-11T23:00:11.1316865Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1317195Z Ran 1 test in 4.358s 2023-01-11T23:00:11.1317336Z 2023-01-11T23:00:11.1317446Z OK (skipped=1) 2023-01-11T23:00:11.1317599Z 2023-01-11T23:00:11.1317733Z Generating XML reports... 2023-01-11T23:00:11.1318382Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111224127.xml 2023-01-11T23:00:11.1319105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1319556Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1320122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1320591Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1321032Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_pwh7npm 2023-01-11T23:00:11.1321572Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_pwh7npm/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1321930Z 2023-01-11T23:00:11.1322043Z Running tests... 2023-01-11T23:00:11.1322428Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1322999Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1323559Z test_basic_nccl_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1324067Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82123 2023-01-11T23:00:11.1324493Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82124 2023-01-11T23:00:11.1325094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1325544Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1326115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1326569Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1327147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1327587Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1328187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1328658Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1329120Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4u62dytx 2023-01-11T23:00:11.1329658Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4u62dytx/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1330168Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0zhcsp46 2023-01-11T23:00:11.1330915Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0zhcsp46/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1331418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1331888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1332258Z skip: Need at least 4 CUDA devices (4.330s) 2023-01-11T23:00:11.1332448Z 2023-01-11T23:00:11.1332732Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1333061Z Ran 1 test in 4.330s 2023-01-11T23:00:11.1333217Z 2023-01-11T23:00:11.1359953Z OK (skipped=1) 2023-01-11T23:00:11.1360169Z 2023-01-11T23:00:11.1360315Z Generating XML reports... 2023-01-11T23:00:11.1361025Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111224134.xml 2023-01-11T23:00:11.1361783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1362235Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1362794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1363259Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1363721Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7_a6cdnb 2023-01-11T23:00:11.1364240Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7_a6cdnb/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1364535Z 2023-01-11T23:00:11.1364640Z Running tests... 2023-01-11T23:00:11.1365039Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1365605Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1366365Z test_async_execution_nested_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1366906Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82226 2023-01-11T23:00:11.1367352Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82227 2023-01-11T23:00:11.1367775Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 82228 2023-01-11T23:00:11.1368217Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 82229 2023-01-11T23:00:11.1368820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1369262Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1369813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1370276Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1371086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1371522Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1372063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1372600Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1373176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1373591Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1374158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1374612Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1375176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1375590Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1376151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1376607Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1377050Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpedyjuqdk 2023-01-11T23:00:11.1377583Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpedyjuqdk/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1378111Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1x2c1woz 2023-01-11T23:00:11.1378637Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1x2c1woz/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1379144Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppw_6be0d 2023-01-11T23:00:11.1379665Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppw_6be0d/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1380188Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphcnzw3ej 2023-01-11T23:00:11.1380713Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphcnzw3ej/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1381196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.1381659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1382116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.1382558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1382926Z fi_getinfo: -61 2023-01-11T23:00:11.1383260Z fi_getinfo: -61 2023-01-11T23:00:11.1383502Z fi_getinfo: -61 2023-01-11T23:00:11.1383768Z fi_getinfo: -61 2023-01-11T23:00:11.1383998Z ok (11.929s) 2023-01-11T23:00:11.1384144Z 2023-01-11T23:00:11.1384393Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1384715Z Ran 1 test in 11.929s 2023-01-11T23:00:11.1384872Z 2023-01-11T23:00:11.1384964Z OK 2023-01-11T23:00:11.1385094Z 2023-01-11T23:00:11.1385214Z Generating XML reports... 2023-01-11T23:00:11.1385895Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224142.xml 2023-01-11T23:00:11.1386676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1387119Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1387684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1388134Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1388591Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpja_4bozn 2023-01-11T23:00:11.1389117Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpja_4bozn/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1389410Z 2023-01-11T23:00:11.1389502Z Running tests... 2023-01-11T23:00:11.1389955Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1390532Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1391120Z test_async_execution_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1391635Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82573 2023-01-11T23:00:11.1392076Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82574 2023-01-11T23:00:11.1392524Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 82575 2023-01-11T23:00:11.1392945Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 82576 2023-01-11T23:00:11.1393550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1393993Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1394562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1395007Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1395577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1396012Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1396558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1397015Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1397581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1398010Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1398554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1399008Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1399569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1399999Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1400544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1401062Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1401518Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmozrgr4x 2023-01-11T23:00:11.1402033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmozrgr4x/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1402560Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeirim45l 2023-01-11T23:00:11.1403072Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphoya74tl 2023-01-11T23:00:11.1403595Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeirim45l/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1404108Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphoya74tl/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1404628Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmk47vmq6 2023-01-11T23:00:11.1405157Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmk47vmq6/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1405659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1406101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1406634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.1407104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.1407472Z fi_getinfo: -61 2023-01-11T23:00:11.1407743Z fi_getinfo: -61 2023-01-11T23:00:11.1408010Z fi_getinfo: -61 2023-01-11T23:00:11.1408262Z fi_getinfo: -61 2023-01-11T23:00:11.1408487Z ok (11.916s) 2023-01-11T23:00:11.1408632Z 2023-01-11T23:00:11.1408894Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1409205Z Ran 1 test in 11.917s 2023-01-11T23:00:11.1409365Z 2023-01-11T23:00:11.1409455Z OK 2023-01-11T23:00:11.1409587Z 2023-01-11T23:00:11.1409712Z Generating XML reports... 2023-01-11T23:00:11.1410707Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224156.xml 2023-01-11T23:00:11.1411490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1411937Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1412504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1412967Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1413406Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6u8lutj9 2023-01-11T23:00:11.1413935Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6u8lutj9/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1414233Z 2023-01-11T23:00:11.1414341Z Running tests... 2023-01-11T23:00:11.1414718Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1415281Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1415876Z test_cuda_future_callback_changes_devices (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1416411Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82920 2023-01-11T23:00:11.1416837Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82921 2023-01-11T23:00:11.1417269Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 82922 2023-01-11T23:00:11.1417704Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 82923 2023-01-11T23:00:11.1418382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1418823Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1419387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1419846Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1420401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1420839Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1421399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1421838Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1422400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1422844Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1423400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1423834Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1424460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1424904Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1425471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1425906Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1426363Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcb4zobum 2023-01-11T23:00:11.1426897Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcb4zobum/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1427406Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppv4rsnqu 2023-01-11T23:00:11.1427934Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppv4rsnqu/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1428460Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6spakqvf 2023-01-11T23:00:11.1428989Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6spakqvf/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1429495Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn224l319 2023-01-11T23:00:11.1430016Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn224l319/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1430513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.1430957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1431417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.1431867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1432203Z ok (11.185s) 2023-01-11T23:00:11.1432333Z 2023-01-11T23:00:11.1432604Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1432932Z Ran 1 test in 11.185s 2023-01-11T23:00:11.1433094Z 2023-01-11T23:00:11.1433186Z OK 2023-01-11T23:00:11.1433317Z 2023-01-11T23:00:11.1433423Z Generating XML reports... 2023-01-11T23:00:11.1434123Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224211.xml 2023-01-11T23:00:11.1434906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1435412Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1435962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1436421Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1436877Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjtd_y1yh 2023-01-11T23:00:11.1437413Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjtd_y1yh/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1437692Z 2023-01-11T23:00:11.1437800Z Running tests... 2023-01-11T23:00:11.1438196Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1438762Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1439347Z test_cuda_future_can_extract_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1439899Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83099 2023-01-11T23:00:11.1440342Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83100 2023-01-11T23:00:11.1440781Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 83101 2023-01-11T23:00:11.1441203Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 83102 2023-01-11T23:00:11.1441855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1442303Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1442854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1443316Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1443882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1444320Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1444864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1445319Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1445884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1446315Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1446856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1447307Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1447870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1448288Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1448849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1449300Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1449753Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxxbfysx2 2023-01-11T23:00:11.1450268Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxxbfysx2/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1451018Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmtjbnxlu 2023-01-11T23:00:11.1451552Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmtjbnxlu/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1452059Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9uu11wch 2023-01-11T23:00:11.1452667Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9uu11wch/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1453186Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp85r0wshh 2023-01-11T23:00:11.1453713Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp85r0wshh/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1454196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.1454666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1455171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1455633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.1455953Z ok (10.117s) 2023-01-11T23:00:11.1456098Z 2023-01-11T23:00:11.1456378Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1456706Z Ran 1 test in 10.117s 2023-01-11T23:00:11.1456862Z 2023-01-11T23:00:11.1456938Z OK 2023-01-11T23:00:11.1457071Z 2023-01-11T23:00:11.1457194Z Generating XML reports... 2023-01-11T23:00:11.1457891Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224225.xml 2023-01-11T23:00:11.1458738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1459177Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1459748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1460208Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1460644Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4q9i7aaw 2023-01-11T23:00:11.1461174Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4q9i7aaw/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1461467Z 2023-01-11T23:00:11.1461575Z Running tests... 2023-01-11T23:00:11.1461969Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1462519Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1462841Z test_cuda_future_can_extract_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1463061Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83334 2023-01-11T23:00:11.1463276Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83335 2023-01-11T23:00:11.1463486Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 83336 2023-01-11T23:00:11.1463694Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 83337 2023-01-11T23:00:11.1464071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1464227Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1464607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1464797Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1465165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1465335Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1465703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1465891Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1466250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1466540Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1466892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1467079Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1467445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1467615Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1467982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1468165Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1468417Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6vfcc5ut 2023-01-11T23:00:11.1468687Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6vfcc5ut/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1468923Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5dfey60w 2023-01-11T23:00:11.1469190Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5dfey60w/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1469488Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcwpymsew 2023-01-11T23:00:11.1469760Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcwpymsew/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1470062Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0u42xlm9 2023-01-11T23:00:11.1470322Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0u42xlm9/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1470547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.1470775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1470997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1471203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.1471306Z ok (10.017s) 2023-01-11T23:00:11.1471325Z 2023-01-11T23:00:11.1471601Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1471714Z Ran 1 test in 10.017s 2023-01-11T23:00:11.1471733Z 2023-01-11T23:00:11.1471824Z OK 2023-01-11T23:00:11.1471843Z 2023-01-11T23:00:11.1471969Z Generating XML reports... 2023-01-11T23:00:11.1472517Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224238.xml 2023-01-11T23:00:11.1472884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1473046Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1473425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1473614Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1473865Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpflaes3j8 2023-01-11T23:00:11.1474134Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpflaes3j8/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1474154Z 2023-01-11T23:00:11.1474260Z Running tests... 2023-01-11T23:00:11.1474523Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1474878Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1475229Z test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1475493Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83509 2023-01-11T23:00:11.1475708Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83510 2023-01-11T23:00:11.1475919Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 83511 2023-01-11T23:00:11.1476130Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 83512 2023-01-11T23:00:11.1476501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1476672Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1477051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1477239Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1477605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1477758Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1478126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1478312Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1478715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1478891Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1479259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1479443Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1479806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1479960Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1480326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1480509Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1480764Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6sxu59d9 2023-01-11T23:00:11.1481032Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6sxu59d9/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1481283Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo1r0nhtx 2023-01-11T23:00:11.1481547Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo1r0nhtx/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1481796Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfxsm22qp 2023-01-11T23:00:11.1482062Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfxsm22qp/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1482291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk7qxqp1v 2023-01-11T23:00:11.1482551Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk7qxqp1v/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1482781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.1483006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1483228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1483448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.1483548Z ok (9.884s) 2023-01-11T23:00:11.1483568Z 2023-01-11T23:00:11.1483834Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1484005Z Ran 1 test in 9.884s 2023-01-11T23:00:11.1484024Z 2023-01-11T23:00:11.1484116Z OK 2023-01-11T23:00:11.1484135Z 2023-01-11T23:00:11.1484258Z Generating XML reports... 2023-01-11T23:00:11.1484808Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224250.xml 2023-01-11T23:00:11.1485176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1485347Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1485724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1485912Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1486161Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpok2we4fy 2023-01-11T23:00:11.1486412Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpok2we4fy/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1486431Z 2023-01-11T23:00:11.1486538Z Running tests... 2023-01-11T23:00:11.1486798Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1487153Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1487535Z test_cuda_future_can_extract_custom_class_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1487758Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83748 2023-01-11T23:00:11.1487971Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83749 2023-01-11T23:00:11.1488182Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 83750 2023-01-11T23:00:11.1488372Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 83751 2023-01-11T23:00:11.1488745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1488922Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1489297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1489488Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1489849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1490019Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1490611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1490804Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1491155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1491325Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1491692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1491876Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1492241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1492408Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1492771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1492952Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1493270Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpujmtryfe 2023-01-11T23:00:11.1493540Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpujmtryfe/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1493791Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5eyedlf7 2023-01-11T23:00:11.1494058Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5eyedlf7/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1494310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0b7g9f0_ 2023-01-11T23:00:11.1494572Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0b7g9f0_/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1494820Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw8ajun9o 2023-01-11T23:00:11.1495081Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw8ajun9o/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1495305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1495517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.1495742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1495966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.1496066Z ok (9.917s) 2023-01-11T23:00:11.1496085Z 2023-01-11T23:00:11.1496426Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1496545Z Ran 1 test in 9.917s 2023-01-11T23:00:11.1496565Z 2023-01-11T23:00:11.1496656Z OK 2023-01-11T23:00:11.1496674Z 2023-01-11T23:00:11.1496797Z Generating XML reports... 2023-01-11T23:00:11.1497326Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224303.xml 2023-01-11T23:00:11.1497692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1497870Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1498247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1498436Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1498690Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0thggrzd 2023-01-11T23:00:11.1498958Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0thggrzd/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1498978Z 2023-01-11T23:00:11.1499085Z Running tests... 2023-01-11T23:00:11.1499345Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1499682Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1500026Z test_cuda_future_can_extract_list_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1500240Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83927 2023-01-11T23:00:11.1500454Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83928 2023-01-11T23:00:11.1500666Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 83929 2023-01-11T23:00:11.1500875Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 83930 2023-01-11T23:00:11.1501246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1501418Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1501795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1501965Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1502383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1502556Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1502923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1503116Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1503476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1503645Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1504012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1504179Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1504548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1504717Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1505081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1505264Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1505565Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr573hwt6 2023-01-11T23:00:11.1505839Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr573hwt6/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1506091Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphwu4wxb3 2023-01-11T23:00:11.1506356Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphwu4wxb3/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1506590Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ck44kub 2023-01-11T23:00:11.1506850Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ck44kub/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1507095Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqbjouk9h 2023-01-11T23:00:11.1507357Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqbjouk9h/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1507586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1507813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.1508034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1508255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.1508340Z ok (10.016s) 2023-01-11T23:00:11.1508378Z 2023-01-11T23:00:11.1508631Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1508744Z Ran 1 test in 10.016s 2023-01-11T23:00:11.1508763Z 2023-01-11T23:00:11.1508854Z OK 2023-01-11T23:00:11.1508873Z 2023-01-11T23:00:11.1508995Z Generating XML reports... 2023-01-11T23:00:11.1509541Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224316.xml 2023-01-11T23:00:11.1509909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1510081Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1510456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1510626Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1510875Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz3k4jyb3 2023-01-11T23:00:11.1511195Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz3k4jyb3/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1511214Z 2023-01-11T23:00:11.1511322Z Running tests... 2023-01-11T23:00:11.1511585Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1511941Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1512272Z test_cuda_future_can_extract_list_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1512487Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84162 2023-01-11T23:00:11.1512699Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84163 2023-01-11T23:00:11.1512895Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 84164 2023-01-11T23:00:11.1513105Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 84165 2023-01-11T23:00:11.1513475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1513647Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1514022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1514260Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1514627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1514795Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1515145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1515332Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1515691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1515859Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1516224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1516411Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1516775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1516941Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1517307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1517475Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1517733Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn4vqb6fu 2023-01-11T23:00:11.1517997Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn4vqb6fu/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1518252Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu2qkxv_n 2023-01-11T23:00:11.1518518Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu2qkxv_n/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1518769Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqahfu0h9 2023-01-11T23:00:11.1519032Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqahfu0h9/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1519278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsastefhi 2023-01-11T23:00:11.1519520Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsastefhi/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1519800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1520025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.1520248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1520471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.1520571Z ok (9.863s) 2023-01-11T23:00:11.1520594Z 2023-01-11T23:00:11.1520862Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1520974Z Ran 1 test in 9.863s 2023-01-11T23:00:11.1520993Z 2023-01-11T23:00:11.1521085Z OK 2023-01-11T23:00:11.1521104Z 2023-01-11T23:00:11.1521210Z Generating XML reports... 2023-01-11T23:00:11.1521755Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224329.xml 2023-01-11T23:00:11.1522126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1522298Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1522676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1522865Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1523170Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4jw0bcdd 2023-01-11T23:00:11.1523444Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4jw0bcdd/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1523464Z 2023-01-11T23:00:11.1523555Z Running tests... 2023-01-11T23:00:11.1523820Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1524175Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1524488Z test_cuda_future_device_as_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1524707Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84337 2023-01-11T23:00:11.1524921Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84338 2023-01-11T23:00:11.1525132Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 84339 2023-01-11T23:00:11.1525344Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 84340 2023-01-11T23:00:11.1525714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1525870Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1526242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1526432Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1526791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1526963Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1527329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1527517Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1527874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1528026Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1528391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1528573Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1528991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1529161Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1529526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1529712Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1529966Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp63t40j_2 2023-01-11T23:00:11.1530231Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp63t40j_2/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1530677Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi7trrf2q 2023-01-11T23:00:11.1530951Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi7trrf2q/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1531206Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1pp5tzlm 2023-01-11T23:00:11.1531471Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1pp5tzlm/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1531723Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwngxvrgh 2023-01-11T23:00:11.1531987Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwngxvrgh/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1532285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.1532518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1532725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.1532949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1533050Z ok (4.520s) 2023-01-11T23:00:11.1533072Z 2023-01-11T23:00:11.1533344Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1533455Z Ran 1 test in 4.521s 2023-01-11T23:00:11.1533475Z 2023-01-11T23:00:11.1533566Z OK 2023-01-11T23:00:11.1533585Z 2023-01-11T23:00:11.1533707Z Generating XML reports... 2023-01-11T23:00:11.1534253Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224341.xml 2023-01-11T23:00:11.1534620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1534775Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1535151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1535338Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1535591Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprqunmk51 2023-01-11T23:00:11.1535855Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprqunmk51/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1535875Z 2023-01-11T23:00:11.1535981Z Running tests... 2023-01-11T23:00:11.1536243Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1536602Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1536906Z test_cuda_future_device_as_int (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1537107Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84508 2023-01-11T23:00:11.1537320Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84509 2023-01-11T23:00:11.1537533Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 84510 2023-01-11T23:00:11.1537814Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 84511 2023-01-11T23:00:11.1538187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1538360Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1538741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1538934Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1539278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1539450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1539819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1540006Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1540370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1540538Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1540902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1541131Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1541508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1541660Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1542025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1542209Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1542471Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpshqqpfcy 2023-01-11T23:00:11.1542743Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpshqqpfcy/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1542996Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbjub9r_a 2023-01-11T23:00:11.1543261Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbjub9r_a/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1543510Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpimqeuns0 2023-01-11T23:00:11.1543756Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpimqeuns0/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1544002Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3jbvqcl8 2023-01-11T23:00:11.1544263Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3jbvqcl8/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1544494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.1544717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1544941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.1545165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1545264Z ok (4.502s) 2023-01-11T23:00:11.1545286Z 2023-01-11T23:00:11.1545553Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1545648Z Ran 1 test in 4.502s 2023-01-11T23:00:11.1545667Z 2023-01-11T23:00:11.1545757Z OK 2023-01-11T23:00:11.1545775Z 2023-01-11T23:00:11.1545896Z Generating XML reports... 2023-01-11T23:00:11.1546444Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224348.xml 2023-01-11T23:00:11.1546869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1547044Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1547417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1547606Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1547841Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw_xj2uv9 2023-01-11T23:00:11.1548104Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw_xj2uv9/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1548124Z 2023-01-11T23:00:11.1548231Z Running tests... 2023-01-11T23:00:11.1548491Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1548846Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1549154Z test_cuda_future_device_as_str (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1549371Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84679 2023-01-11T23:00:11.1549584Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84680 2023-01-11T23:00:11.1549798Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 84681 2023-01-11T23:00:11.1550034Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 84682 2023-01-11T23:00:11.1550412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1550583Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1550958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1551149Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1551509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1551678Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1552044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1552219Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1552576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1552746Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1553113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1553297Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1553666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1553835Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1554201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1554384Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1554625Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpakmpmlkx 2023-01-11T23:00:11.1554935Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpakmpmlkx/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1555189Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo1p6ys5n 2023-01-11T23:00:11.1555453Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo1p6ys5n/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1555758Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc67dwss2 2023-01-11T23:00:11.1556020Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc67dwss2/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1556267Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp83_jffc3 2023-01-11T23:00:11.1556527Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp83_jffc3/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1556740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1556967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.1557189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1557483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.1557583Z ok (4.368s) 2023-01-11T23:00:11.1557605Z 2023-01-11T23:00:11.1557874Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1557986Z Ran 1 test in 4.368s 2023-01-11T23:00:11.1558006Z 2023-01-11T23:00:11.1558098Z OK 2023-01-11T23:00:11.1558117Z 2023-01-11T23:00:11.1558238Z Generating XML reports... 2023-01-11T23:00:11.1558764Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224356.xml 2023-01-11T23:00:11.1559188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1559368Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1559746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1559935Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1560189Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfgklmeps 2023-01-11T23:00:11.1560455Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfgklmeps/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1560474Z 2023-01-11T23:00:11.1560584Z Running tests... 2023-01-11T23:00:11.1560826Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1561186Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1561492Z test_cuda_future_device_not_cuda (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1561706Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84850 2023-01-11T23:00:11.1561920Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84851 2023-01-11T23:00:11.1562133Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 84852 2023-01-11T23:00:11.1562344Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 84853 2023-01-11T23:00:11.1562711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1562882Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1563240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1563434Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1563793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1563961Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1564332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1564519Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1564930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1565100Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1565449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1565636Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1565996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1566163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1566529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1566712Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1566968Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphgoft2qo 2023-01-11T23:00:11.1567235Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphgoft2qo/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1567486Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3afbazmy 2023-01-11T23:00:11.1567735Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3afbazmy/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1568026Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9fs394z7 2023-01-11T23:00:11.1568295Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9fs394z7/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1568545Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp6wjj0vk 2023-01-11T23:00:11.1568806Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp6wjj0vk/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1569036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.1569261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1569484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1569707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.1569791Z ok (4.509s) 2023-01-11T23:00:11.1569813Z 2023-01-11T23:00:11.1570081Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1570193Z Ran 1 test in 4.509s 2023-01-11T23:00:11.1570212Z 2023-01-11T23:00:11.1570508Z OK 2023-01-11T23:00:11.1570529Z 2023-01-11T23:00:11.1570661Z Generating XML reports... 2023-01-11T23:00:11.1571213Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224403.xml 2023-01-11T23:00:11.1571582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1571754Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1572112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1572301Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1572557Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxi5hzrpa 2023-01-11T23:00:11.1572826Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxi5hzrpa/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1572847Z 2023-01-11T23:00:11.1572952Z Running tests... 2023-01-11T23:00:11.1573213Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1573568Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1573971Z test_cuda_future_modify_tensor_inplace (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1574186Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85021 2023-01-11T23:00:11.1574384Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85022 2023-01-11T23:00:11.1574594Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 85023 2023-01-11T23:00:11.1574807Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 85024 2023-01-11T23:00:11.1575180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1575352Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1575728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1575920Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1576280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1576432Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1576803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1577052Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1577427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1577597Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1577965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1578149Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1578515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1578685Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1579032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1579220Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1579475Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd331w1if 2023-01-11T23:00:11.1579741Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd331w1if/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1579994Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppc8w0rmw 2023-01-11T23:00:11.1580262Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppc8w0rmw/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1580516Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiy66dxhk 2023-01-11T23:00:11.1580781Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiy66dxhk/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1581013Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ylnab8p 2023-01-11T23:00:11.1581277Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ylnab8p/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1581505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1581732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1581956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.1582181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.1582336Z ok (6.123s) 2023-01-11T23:00:11.1582355Z 2023-01-11T23:00:11.1582622Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1582734Z Ran 1 test in 6.123s 2023-01-11T23:00:11.1582753Z 2023-01-11T23:00:11.1582828Z OK 2023-01-11T23:00:11.1582847Z 2023-01-11T23:00:11.1582970Z Generating XML reports... 2023-01-11T23:00:11.1583515Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224410.xml 2023-01-11T23:00:11.1583883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1584058Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1584438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1584626Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1584881Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjdh0hzmu 2023-01-11T23:00:11.1585132Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjdh0hzmu/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1585168Z 2023-01-11T23:00:11.1585259Z Running tests... 2023-01-11T23:00:11.1585521Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1585922Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1586234Z test_cuda_future_replace_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1586450Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85196 2023-01-11T23:00:11.1586666Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85197 2023-01-11T23:00:11.1586878Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 85198 2023-01-11T23:00:11.1587091Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 85199 2023-01-11T23:00:11.1587446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1587616Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1587993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1588183Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1588544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1588715Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1589085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1589274Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1589616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1589786Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1590152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1590344Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1590708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1590876Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1591241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1591423Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1591730Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp33k157h0 2023-01-11T23:00:11.1591978Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp33k157h0/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1592231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5l7b5xdp 2023-01-11T23:00:11.1592497Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5l7b5xdp/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1592752Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpedz4spu6 2023-01-11T23:00:11.1593017Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpedz4spu6/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1593265Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp51hl430q 2023-01-11T23:00:11.1593523Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp51hl430q/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1593751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.1593976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1594182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.1594402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1594502Z ok (6.085s) 2023-01-11T23:00:11.1594566Z 2023-01-11T23:00:11.1594838Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1594948Z Ran 1 test in 6.085s 2023-01-11T23:00:11.1594968Z 2023-01-11T23:00:11.1595060Z OK 2023-01-11T23:00:11.1595078Z 2023-01-11T23:00:11.1595200Z Generating XML reports... 2023-01-11T23:00:11.1595747Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224419.xml 2023-01-11T23:00:11.1596102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1596276Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1596655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1596843Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1597096Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa0jqm52s 2023-01-11T23:00:11.1597360Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa0jqm52s/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1597380Z 2023-01-11T23:00:11.1597486Z Running tests... 2023-01-11T23:00:11.1597747Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1598102Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1598400Z test_cuda_future_value_on_bad_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1598615Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85371 2023-01-11T23:00:11.1598828Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85372 2023-01-11T23:00:11.1599038Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 85373 2023-01-11T23:00:11.1599250Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 85374 2023-01-11T23:00:11.1599619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1599791Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1600164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1600398Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1600763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1600933Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1601303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1601492Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1601851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1602020Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1602387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1602571Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1602922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1603091Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1603455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1603748Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1604013Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpumcuhpf9 2023-01-11T23:00:11.1604282Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpumcuhpf9/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1604533Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc5irsa5j 2023-01-11T23:00:11.1604796Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc5irsa5j/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1605034Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj72xirqf 2023-01-11T23:00:11.1605299Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj72xirqf/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1605545Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpubwscnyr 2023-01-11T23:00:11.1605805Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpubwscnyr/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1606033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.1606259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1606483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1606704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.1606807Z ok (11.281s) 2023-01-11T23:00:11.1606827Z 2023-01-11T23:00:11.1607077Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1607191Z Ran 1 test in 11.281s 2023-01-11T23:00:11.1607210Z 2023-01-11T23:00:11.1607303Z OK 2023-01-11T23:00:11.1607322Z 2023-01-11T23:00:11.1607443Z Generating XML reports... 2023-01-11T23:00:11.1607991Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224428.xml 2023-01-11T23:00:11.1608357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1608530Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1608908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1609079Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1609382Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpavsownqb 2023-01-11T23:00:11.1609650Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpavsownqb/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1609670Z 2023-01-11T23:00:11.1609777Z Running tests... 2023-01-11T23:00:11.1610040Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1610655Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1610954Z test_custom_stream (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1611729Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/79750 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.600s) 2023-01-11T23:00:11.1611753Z 2023-01-11T23:00:11.1612014Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1612126Z Ran 1 test in 1.601s 2023-01-11T23:00:11.1612145Z 2023-01-11T23:00:11.1612235Z OK (skipped=1) 2023-01-11T23:00:11.1612254Z 2023-01-11T23:00:11.1612383Z Generating XML reports... 2023-01-11T23:00:11.1612997Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224442.xml 2023-01-11T23:00:11.1613378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1613551Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1613926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1614116Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1614373Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpai_3r6e0 2023-01-11T23:00:11.1614639Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpai_3r6e0/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1614659Z 2023-01-11T23:00:11.1614750Z Running tests... 2023-01-11T23:00:11.1615012Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1615370Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1615665Z test_custom_stream_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1615882Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85584 2023-01-11T23:00:11.1616098Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85585 2023-01-11T23:00:11.1616311Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 85586 2023-01-11T23:00:11.1616522Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 85587 2023-01-11T23:00:11.1616877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1617051Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1617430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1617626Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1617986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1618157Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1618527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1618796Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1619163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1619316Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1619680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1619872Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1620461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1620913Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1621454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1621926Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1622392Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf1r5hnkw 2023-01-11T23:00:11.1622929Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf1r5hnkw/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1623441Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph9xpllvf 2023-01-11T23:00:11.1623977Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph9xpllvf/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1624562Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu29aiatj 2023-01-11T23:00:11.1625083Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu29aiatj/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1625609Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdrrjx8vh 2023-01-11T23:00:11.1626133Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdrrjx8vh/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1626646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1627095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1627562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.1628022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.1628405Z fi_getinfo: -61 2023-01-11T23:00:11.1628668Z fi_getinfo: -61 2023-01-11T23:00:11.1628936Z fi_getinfo: -61 2023-01-11T23:00:11.1629207Z fi_getinfo: -61 2023-01-11T23:00:11.1629422Z ok (20.423s) 2023-01-11T23:00:11.1629568Z 2023-01-11T23:00:11.1629833Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1630157Z Ran 1 test in 20.423s 2023-01-11T23:00:11.1630296Z 2023-01-11T23:00:11.1630388Z OK 2023-01-11T23:00:11.1630520Z 2023-01-11T23:00:11.1630644Z Generating XML reports... 2023-01-11T23:00:11.1631352Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224446.xml 2023-01-11T23:00:11.1632129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1632580Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1633146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1633614Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1634051Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw3cd3cgq 2023-01-11T23:00:11.1634590Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw3cd3cgq/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1634892Z 2023-01-11T23:00:11.1635008Z Running tests... 2023-01-11T23:00:11.1635479Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1636071Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1636645Z test_custom_stream_nested (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1637164Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85943 2023-01-11T23:00:11.1637609Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85944 2023-01-11T23:00:11.1638034Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 85945 2023-01-11T23:00:11.1638464Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 85946 2023-01-11T23:00:11.1639060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1639501Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1640050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1640508Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1641076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1641492Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1642101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1642564Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1643133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1643550Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1644109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1644627Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1645198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1645638Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1646194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1646652Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1647100Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfkn7vtdn 2023-01-11T23:00:11.1647654Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfkn7vtdn/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1648179Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6vcof7o_ 2023-01-11T23:00:11.1648713Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6vcof7o_/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1649233Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn2o8ay__ 2023-01-11T23:00:11.1649737Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn2o8ay__/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1650258Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw0qx_q5j 2023-01-11T23:00:11.1651006Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw0qx_q5j/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1651488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.1651954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.1652408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1652958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1653332Z fi_getinfo: -61 2023-01-11T23:00:11.1653607Z fi_getinfo: -61 2023-01-11T23:00:11.1653878Z fi_getinfo: -61 2023-01-11T23:00:11.1654131Z fi_getinfo: -61 2023-01-11T23:00:11.1654362Z ok (13.422s) 2023-01-11T23:00:11.1654508Z 2023-01-11T23:00:11.1654775Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1655131Z Ran 1 test in 13.422s 2023-01-11T23:00:11.1655290Z 2023-01-11T23:00:11.1655384Z OK 2023-01-11T23:00:11.1655520Z 2023-01-11T23:00:11.1655645Z Generating XML reports... 2023-01-11T23:00:11.1656365Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224509.xml 2023-01-11T23:00:11.1657159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1657603Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1658167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1658627Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1659065Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1r0ytzg1 2023-01-11T23:00:11.1659662Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1r0ytzg1/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1659964Z 2023-01-11T23:00:11.1660072Z Running tests... 2023-01-11T23:00:11.1660472Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1661018Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1661592Z test_custom_stream_nested_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1662119Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86302 2023-01-11T23:00:11.1662546Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86303 2023-01-11T23:00:11.1662982Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 86304 2023-01-11T23:00:11.1663420Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 86305 2023-01-11T23:00:11.1664054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1664499Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1665072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1665554Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1666127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1666560Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1669426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1669935Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1670545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1670973Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1671538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1671996Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1672568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1673080Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1673644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1674106Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1674542Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg_yw8wky 2023-01-11T23:00:11.1675083Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg_yw8wky/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1675614Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz8ubpxjj 2023-01-11T23:00:11.1676146Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz8ubpxjj/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1676652Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkgs0f4at 2023-01-11T23:00:11.1677185Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkgs0f4at/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1677708Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuijq0bd4 2023-01-11T23:00:11.1678215Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuijq0bd4/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1678717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.1679240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.1679716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1680160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1680540Z fi_getinfo: -61 2023-01-11T23:00:11.1680815Z fi_getinfo: -61 2023-01-11T23:00:11.1681068Z fi_getinfo: -61 2023-01-11T23:00:11.1681340Z fi_getinfo: -61 2023-01-11T23:00:11.1681580Z ok (11.711s) 2023-01-11T23:00:11.1681729Z 2023-01-11T23:00:11.1681979Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1682305Z Ran 1 test in 11.712s 2023-01-11T23:00:11.1682463Z 2023-01-11T23:00:11.1682560Z OK 2023-01-11T23:00:11.1682694Z 2023-01-11T23:00:11.1682819Z Generating XML reports... 2023-01-11T23:00:11.1683502Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224526.xml 2023-01-11T23:00:11.1684285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1684730Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1685281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1685752Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1686216Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbhkvyrh1 2023-01-11T23:00:11.1686748Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbhkvyrh1/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1687042Z 2023-01-11T23:00:11.1687136Z Running tests... 2023-01-11T23:00:11.1687537Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1688110Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1688677Z test_device_map_cpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1689171Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86656 2023-01-11T23:00:11.1689614Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86657 2023-01-11T23:00:11.1690063Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 86658 2023-01-11T23:00:11.1690941Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 86659 2023-01-11T23:00:11.1691548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1691993Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1692562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1693015Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1693583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1694024Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1694556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1695001Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1695567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1696027Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1696584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1697158Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1697743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1698163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1698722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1699180Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1699642Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdrzyib_5 2023-01-11T23:00:11.1700164Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdrzyib_5/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1700689Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzme1t3wi 2023-01-11T23:00:11.1701219Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzme1t3wi/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1701743Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiitovdrv 2023-01-11T23:00:11.1702252Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiitovdrv/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1702776Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm3ouxcb7 2023-01-11T23:00:11.1703301Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm3ouxcb7/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1703786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.1704251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1704709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.1705165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1705538Z fi_getinfo: -61 2023-01-11T23:00:11.1705813Z fi_getinfo: -61 2023-01-11T23:00:11.1706082Z fi_getinfo: -61 2023-01-11T23:00:11.1706335Z fi_getinfo: -61 2023-01-11T23:00:11.1706565Z ok (4.960s) 2023-01-11T23:00:11.1706712Z 2023-01-11T23:00:11.1706977Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1707283Z Ran 1 test in 4.960s 2023-01-11T23:00:11.1707443Z 2023-01-11T23:00:11.1707539Z OK 2023-01-11T23:00:11.1707670Z 2023-01-11T23:00:11.1707871Z Generating XML reports... 2023-01-11T23:00:11.1708575Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224540.xml 2023-01-11T23:00:11.1709337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1709779Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1710347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1710797Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1711256Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9utxzsba 2023-01-11T23:00:11.1711787Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9utxzsba/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1712086Z 2023-01-11T23:00:11.1712195Z Running tests... 2023-01-11T23:00:11.1712583Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1713144Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1713726Z test_device_map_cpu_to_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1714238Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86999 2023-01-11T23:00:11.1714732Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87000 2023-01-11T23:00:11.1715176Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 87001 2023-01-11T23:00:11.1715613Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 87002 2023-01-11T23:00:11.1716194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1716638Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1717209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1717675Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1718227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1718665Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1719225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1719664Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1720230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1720662Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1721227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1721665Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1722235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1722672Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1723215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1723677Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1724131Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4qz2800c 2023-01-11T23:00:11.1724657Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4qz2800c/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1725161Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyn_a0xdu 2023-01-11T23:00:11.1725749Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyn_a0xdu/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1726276Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw7dc59l0 2023-01-11T23:00:11.1726803Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw7dc59l0/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1727306Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxqmz0fjk 2023-01-11T23:00:11.1727829Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxqmz0fjk/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1728327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.1728773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1729231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1729687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.1730074Z fi_getinfo: -61 2023-01-11T23:00:11.1730543Z fi_getinfo: -61 2023-01-11T23:00:11.1730826Z fi_getinfo: -61 2023-01-11T23:00:11.1731099Z fi_getinfo: -61 2023-01-11T23:00:11.1731311Z ok (7.921s) 2023-01-11T23:00:11.1731458Z 2023-01-11T23:00:11.1731724Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1732141Z Ran 1 test in 7.921s 2023-01-11T23:00:11.1732310Z 2023-01-11T23:00:11.1732386Z OK 2023-01-11T23:00:11.1732519Z 2023-01-11T23:00:11.1732643Z Generating XML reports... 2023-01-11T23:00:11.1733344Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224548.xml 2023-01-11T23:00:11.1734125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1734558Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1735125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1735589Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1736028Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0yotqeit 2023-01-11T23:00:11.1736565Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0yotqeit/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1736864Z 2023-01-11T23:00:11.1736974Z Running tests... 2023-01-11T23:00:11.1737373Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1737921Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1738508Z test_device_map_cpu_to_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1739047Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87350 2023-01-11T23:00:11.1739471Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87351 2023-01-11T23:00:11.1739913Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 87352 2023-01-11T23:00:11.1740346Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 87353 2023-01-11T23:00:11.1740953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1741380Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1741949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1742409Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1742963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1743475Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1744039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1744498Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1745052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1745493Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1746053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1746510Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1747061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1747501Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1748056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1748492Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1748998Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpej0i0hpz 2023-01-11T23:00:11.1749540Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpej0i0hpz/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1750067Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa1i8j1aa 2023-01-11T23:00:11.1750574Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa1i8j1aa/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1751096Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb3c7iygj 2023-01-11T23:00:11.1751626Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb3c7iygj/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1752145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzm_l70d_ 2023-01-11T23:00:11.1752645Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzm_l70d_/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1753143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.1753612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.1754053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1754504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1754944Z fi_getinfo: -61 2023-01-11T23:00:11.1755201Z fi_getinfo: -61 2023-01-11T23:00:11.1755474Z fi_getinfo: -61 2023-01-11T23:00:11.1755746Z fi_getinfo: -61 2023-01-11T23:00:11.1755961Z ok (7.926s) 2023-01-11T23:00:11.1756106Z 2023-01-11T23:00:11.1756376Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1756697Z Ran 1 test in 7.926s 2023-01-11T23:00:11.1756859Z 2023-01-11T23:00:11.1756953Z OK 2023-01-11T23:00:11.1757069Z 2023-01-11T23:00:11.1757193Z Generating XML reports... 2023-01-11T23:00:11.1757889Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224558.xml 2023-01-11T23:00:11.1758669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1759112Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1759661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1760120Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1760637Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcon259pd 2023-01-11T23:00:11.1761153Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcon259pd/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1761449Z 2023-01-11T23:00:11.1761559Z Running tests... 2023-01-11T23:00:11.1761965Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1762537Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1763096Z test_device_map_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1763619Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87701 2023-01-11T23:00:11.1764063Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87702 2023-01-11T23:00:11.1764481Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 87703 2023-01-11T23:00:11.1764923Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 87704 2023-01-11T23:00:11.1765519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1765964Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1766559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1767026Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1767598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1768015Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1768578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1769038Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1769603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1770019Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1770808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1771270Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1771842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1772264Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1772819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1773276Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1773717Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw31kjc9a 2023-01-11T23:00:11.1774249Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw31kjc9a/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1774769Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8hg0h2ny 2023-01-11T23:00:11.1775297Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8hg0h2ny/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1775799Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1i95l6bo 2023-01-11T23:00:11.1776321Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1i95l6bo/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1776836Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbytax2q7 2023-01-11T23:00:11.1777344Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbytax2q7/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1777932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.1778396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.1778856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1779300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1779684Z fi_getinfo: -61 2023-01-11T23:00:11.1779963Z fi_getinfo: -61 2023-01-11T23:00:11.1780214Z fi_getinfo: -61 2023-01-11T23:00:11.1780484Z fi_getinfo: -61 2023-01-11T23:00:11.1780720Z ok (7.916s) 2023-01-11T23:00:11.1780865Z 2023-01-11T23:00:11.1781113Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1781439Z Ran 1 test in 7.916s 2023-01-11T23:00:11.1781597Z 2023-01-11T23:00:11.1781691Z OK 2023-01-11T23:00:11.1781828Z 2023-01-11T23:00:11.1781953Z Generating XML reports... 2023-01-11T23:00:11.1782632Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224609.xml 2023-01-11T23:00:11.1783417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1783863Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1784478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1784952Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1785413Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqq9y2x19 2023-01-11T23:00:11.1785952Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqq9y2x19/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1786242Z 2023-01-11T23:00:11.1786338Z Running tests... 2023-01-11T23:00:11.1786746Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1787313Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1787905Z test_device_map_gpu_default_to_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1788969Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/80008 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.601s) 2023-01-11T23:00:11.1789485Z 2023-01-11T23:00:11.1789748Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1790073Z Ran 1 test in 1.601s 2023-01-11T23:00:11.1790236Z 2023-01-11T23:00:11.1790344Z OK (skipped=1) 2023-01-11T23:00:11.1790483Z 2023-01-11T23:00:11.1790607Z Generating XML reports... 2023-01-11T23:00:11.1791299Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224620.xml 2023-01-11T23:00:11.1792079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1792506Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1793081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1793540Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1794000Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9mdktm3h 2023-01-11T23:00:11.1794510Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9mdktm3h/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1794866Z 2023-01-11T23:00:11.1794975Z Running tests... 2023-01-11T23:00:11.1795378Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1795942Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1796497Z test_device_map_gpu_mixed_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1797031Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88082 2023-01-11T23:00:11.1797477Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88083 2023-01-11T23:00:11.1797898Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 88084 2023-01-11T23:00:11.1798334Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 88085 2023-01-11T23:00:11.1798937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1799386Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1799937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1800401Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1800973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1801442Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1802012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1802466Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1803039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1803458Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1804027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1804483Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1805048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1805470Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1806032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1806491Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1806927Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplddwi_kb 2023-01-11T23:00:11.1807463Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplddwi_kb/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1807991Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyoqif_6j 2023-01-11T23:00:11.1808520Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyoqif_6j/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1809025Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyx72uvw4 2023-01-11T23:00:11.1809552Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyx72uvw4/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1810076Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqhqpbyun 2023-01-11T23:00:11.1810815Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqhqpbyun/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1811321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1811787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1812333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.1812777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.1813168Z fi_getinfo: -61 2023-01-11T23:00:11.1813448Z fi_getinfo: -61 2023-01-11T23:00:11.1813700Z fi_getinfo: -61 2023-01-11T23:00:11.1813970Z fi_getinfo: -61 2023-01-11T23:00:11.1814203Z ok (10.411s) 2023-01-11T23:00:11.1814350Z 2023-01-11T23:00:11.1814602Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1814929Z Ran 1 test in 10.411s 2023-01-11T23:00:11.1815088Z 2023-01-11T23:00:11.1815181Z OK 2023-01-11T23:00:11.1815315Z 2023-01-11T23:00:11.1815442Z Generating XML reports... 2023-01-11T23:00:11.1816120Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224624.xml 2023-01-11T23:00:11.1816898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1817349Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1817897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1818362Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1818887Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwo5g9s9z 2023-01-11T23:00:11.1819430Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwo5g9s9z/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1819729Z 2023-01-11T23:00:11.1819821Z Running tests... 2023-01-11T23:00:11.1820225Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1820789Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1821364Z test_device_map_gpu_mixed_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1821872Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88433 2023-01-11T23:00:11.1822315Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88434 2023-01-11T23:00:11.1822757Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 88435 2023-01-11T23:00:11.1823178Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 88436 2023-01-11T23:00:11.1823774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1824220Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1824787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1825234Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1825805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1826244Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1826786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1827244Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1827809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1828239Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1828779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1829235Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1829802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1830292Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1830837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1831289Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1831748Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6b2x3o85 2023-01-11T23:00:11.1832264Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6b2x3o85/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1832788Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8kwjh5tu 2023-01-11T23:00:11.1833315Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8kwjh5tu/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1833835Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpziw3e2if 2023-01-11T23:00:11.1834347Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpziw3e2if/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1834867Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4np71bvx 2023-01-11T23:00:11.1835392Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4np71bvx/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1835920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.1836394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1836854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1837313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.1837680Z fi_getinfo: -61 2023-01-11T23:00:11.1837954Z fi_getinfo: -61 2023-01-11T23:00:11.1838229Z fi_getinfo: -61 2023-01-11T23:00:11.1838485Z fi_getinfo: -61 2023-01-11T23:00:11.1838714Z ok (10.307s) 2023-01-11T23:00:11.1838861Z 2023-01-11T23:00:11.1839129Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1839434Z Ran 1 test in 10.307s 2023-01-11T23:00:11.1839594Z 2023-01-11T23:00:11.1839687Z OK 2023-01-11T23:00:11.1839820Z 2023-01-11T23:00:11.1839944Z Generating XML reports... 2023-01-11T23:00:11.1840643Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224637.xml 2023-01-11T23:00:11.1841403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1841847Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1842413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1842866Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1843330Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb1u1f6y9 2023-01-11T23:00:11.1843863Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb1u1f6y9/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1844158Z 2023-01-11T23:00:11.1844267Z Running tests... 2023-01-11T23:00:11.1844653Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1845225Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1845801Z test_device_map_gpu_mixed_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1846304Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88784 2023-01-11T23:00:11.1846749Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88785 2023-01-11T23:00:11.1847303Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 88786 2023-01-11T23:00:11.1847745Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 88787 2023-01-11T23:00:11.1848329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1848771Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1849337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1849805Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1850565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1851015Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1851580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1852027Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1852593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1853031Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1853669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1854118Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1854726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1855165Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1855706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1856165Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1856621Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb0cc0_qc 2023-01-11T23:00:11.1857154Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb0cc0_qc/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1857661Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp46ykziap 2023-01-11T23:00:11.1858192Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp46ykziap/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1858714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2fnxs4aa 2023-01-11T23:00:11.1859239Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2fnxs4aa/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1859742Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0esstl5c 2023-01-11T23:00:11.1860267Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0esstl5c/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1860764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.1861211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1861667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.1862126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1862515Z fi_getinfo: -61 2023-01-11T23:00:11.1862638Z fi_getinfo: -61 2023-01-11T23:00:11.1862775Z fi_getinfo: -61 2023-01-11T23:00:11.1862912Z fi_getinfo: -61 2023-01-11T23:00:11.1863012Z ok (10.298s) 2023-01-11T23:00:11.1863033Z 2023-01-11T23:00:11.1863302Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1863414Z Ran 1 test in 10.298s 2023-01-11T23:00:11.1863434Z 2023-01-11T23:00:11.1863609Z OK 2023-01-11T23:00:11.1863626Z 2023-01-11T23:00:11.1863734Z Generating XML reports... 2023-01-11T23:00:11.1864282Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224650.xml 2023-01-11T23:00:11.1864649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1864823Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1865209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1865397Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1865646Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjkzyv_a5 2023-01-11T23:00:11.1865912Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjkzyv_a5/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1865934Z 2023-01-11T23:00:11.1866045Z Running tests... 2023-01-11T23:00:11.1866290Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1866646Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1866947Z test_device_map_gpu_mixed_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1867213Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89135 2023-01-11T23:00:11.1867433Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89136 2023-01-11T23:00:11.1867648Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 89137 2023-01-11T23:00:11.1867855Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 89138 2023-01-11T23:00:11.1868229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1868407Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1868767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1868955Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1869318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1869493Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1869868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1870061Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1870415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1870587Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1870932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1871119Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1871487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1871659Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1872023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1872211Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1872465Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaxvq9h6z 2023-01-11T23:00:11.1872737Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaxvq9h6z/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1873041Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7xecuczz 2023-01-11T23:00:11.1873287Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7xecuczz/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1873538Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8be7641y 2023-01-11T23:00:11.1873799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8be7641y/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1874044Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzawcv9n1 2023-01-11T23:00:11.1874312Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzawcv9n1/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1874537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.1874759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1874990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.1875212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1875341Z fi_getinfo: -61 2023-01-11T23:00:11.1875483Z fi_getinfo: -61 2023-01-11T23:00:11.1875620Z fi_getinfo: -61 2023-01-11T23:00:11.1875756Z fi_getinfo: -61 2023-01-11T23:00:11.1875855Z ok (10.476s) 2023-01-11T23:00:11.1875875Z 2023-01-11T23:00:11.1876187Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1876308Z Ran 1 test in 10.476s 2023-01-11T23:00:11.1876327Z 2023-01-11T23:00:11.1876401Z OK 2023-01-11T23:00:11.1876420Z 2023-01-11T23:00:11.1876581Z Generating XML reports... 2023-01-11T23:00:11.1877128Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224703.xml 2023-01-11T23:00:11.1877499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1877676Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1878057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1878248Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1878504Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyhyf6ddb 2023-01-11T23:00:11.1878771Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyhyf6ddb/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1878790Z 2023-01-11T23:00:11.1878882Z Running tests... 2023-01-11T23:00:11.1879147Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1879501Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1879808Z test_device_map_gpu_mixed_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1880024Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89486 2023-01-11T23:00:11.1880238Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89487 2023-01-11T23:00:11.1880450Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 89488 2023-01-11T23:00:11.1880660Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 89489 2023-01-11T23:00:11.1881010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1881187Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1881567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1881757Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1882181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1882354Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1882731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1882915Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1883277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1883431Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1883802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1883991Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1884353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1884523Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1884889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1885078Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1885374Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5lap89d_ 2023-01-11T23:00:11.1885648Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5lap89d_/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1885881Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfbus02ex 2023-01-11T23:00:11.1886146Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfbus02ex/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1886397Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplf5xllkj 2023-01-11T23:00:11.1886660Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplf5xllkj/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1886906Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptw9dm0cs 2023-01-11T23:00:11.1887169Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptw9dm0cs/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1887399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.1887620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1887824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.1888042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1888194Z fi_getinfo: -61 2023-01-11T23:00:11.1888334Z fi_getinfo: -61 2023-01-11T23:00:11.1888478Z fi_getinfo: -61 2023-01-11T23:00:11.1888615Z fi_getinfo: -61 2023-01-11T23:00:11.1888714Z ok (10.273s) 2023-01-11T23:00:11.1888734Z 2023-01-11T23:00:11.1889000Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1889097Z Ran 1 test in 10.273s 2023-01-11T23:00:11.1889116Z 2023-01-11T23:00:11.1889207Z OK 2023-01-11T23:00:11.1889227Z 2023-01-11T23:00:11.1889354Z Generating XML reports... 2023-01-11T23:00:11.1889900Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224717.xml 2023-01-11T23:00:11.1890268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1890691Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1891078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1891351Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1891585Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpujad9ov4 2023-01-11T23:00:11.1891850Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpujad9ov4/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1891870Z 2023-01-11T23:00:11.1891978Z Running tests... 2023-01-11T23:00:11.1892250Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1892606Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1892904Z test_device_map_gpu_mixed_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1893120Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89845 2023-01-11T23:00:11.1893336Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89846 2023-01-11T23:00:11.1893550Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 89847 2023-01-11T23:00:11.1893741Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 89848 2023-01-11T23:00:11.1894115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1894288Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1894723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1894921Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1895284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1895457Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1895826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1896017Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1896355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1896532Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1896902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1897087Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1897449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1897617Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1897982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1898167Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1898403Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdi2vsc4w 2023-01-11T23:00:11.1898674Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdi2vsc4w/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1898926Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph9p06570 2023-01-11T23:00:11.1899196Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph9p06570/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1899446Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpinngxxol 2023-01-11T23:00:11.1899712Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpinngxxol/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1899957Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpldazshf3 2023-01-11T23:00:11.1900273Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpldazshf3/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1900501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1900706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.1900933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1901158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.1901308Z fi_getinfo: -61 2023-01-11T23:00:11.1901447Z fi_getinfo: -61 2023-01-11T23:00:11.1901584Z fi_getinfo: -61 2023-01-11T23:00:11.1901724Z fi_getinfo: -61 2023-01-11T23:00:11.1901806Z ok (10.312s) 2023-01-11T23:00:11.1901843Z 2023-01-11T23:00:11.1902088Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1902203Z Ran 1 test in 10.312s 2023-01-11T23:00:11.1902225Z 2023-01-11T23:00:11.1902318Z OK 2023-01-11T23:00:11.1902338Z 2023-01-11T23:00:11.1902465Z Generating XML reports... 2023-01-11T23:00:11.1903014Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224730.xml 2023-01-11T23:00:11.1903378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1903595Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1903980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1904151Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1904400Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp57p3q70k 2023-01-11T23:00:11.1904668Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp57p3q70k/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1904691Z 2023-01-11T23:00:11.1904797Z Running tests... 2023-01-11T23:00:11.1905062Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1905422Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1905721Z test_device_map_gpu_mixed_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1905941Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90204 2023-01-11T23:00:11.1906152Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90205 2023-01-11T23:00:11.1906348Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 90206 2023-01-11T23:00:11.1906557Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 90207 2023-01-11T23:00:11.1906929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1907107Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1907483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1907669Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1908033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1908203Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1908554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1908745Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1909103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1909323Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1909693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1909875Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1910240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1910414Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1910779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1910945Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1911197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyh694n19 2023-01-11T23:00:11.1911466Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyh694n19/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1911714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvr2vszcx 2023-01-11T23:00:11.1911980Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvr2vszcx/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1912228Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp24m_mjwi 2023-01-11T23:00:11.1912537Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp24m_mjwi/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1912784Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprs2a8l3j 2023-01-11T23:00:11.1913026Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprs2a8l3j/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1913255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1913479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.1913707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1913929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.1914076Z fi_getinfo: -61 2023-01-11T23:00:11.1914215Z fi_getinfo: -61 2023-01-11T23:00:11.1914350Z fi_getinfo: -61 2023-01-11T23:00:11.1914470Z fi_getinfo: -61 2023-01-11T23:00:11.1914570Z ok (10.408s) 2023-01-11T23:00:11.1914589Z 2023-01-11T23:00:11.1914857Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1914968Z Ran 1 test in 10.409s 2023-01-11T23:00:11.1914987Z 2023-01-11T23:00:11.1915081Z OK 2023-01-11T23:00:11.1915100Z 2023-01-11T23:00:11.1915226Z Generating XML reports... 2023-01-11T23:00:11.1915768Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224743.xml 2023-01-11T23:00:11.1916135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1916290Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1916667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1916860Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1917113Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcayj7fu5 2023-01-11T23:00:11.1917374Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcayj7fu5/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1917394Z 2023-01-11T23:00:11.1917504Z Running tests... 2023-01-11T23:00:11.1917767Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1918123Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1918474Z test_device_map_gpu_mixed_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1918672Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90563 2023-01-11T23:00:11.1918888Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90564 2023-01-11T23:00:11.1919101Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 90565 2023-01-11T23:00:11.1919312Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 90566 2023-01-11T23:00:11.1919687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1919858Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1920236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1920426Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1920785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1920937Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1921306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1921550Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1921919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1922088Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1922457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1922643Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1923012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1923163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1923525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1923711Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1923962Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwovc39ga 2023-01-11T23:00:11.1924226Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwovc39ga/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1924475Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpevx_1rs_ 2023-01-11T23:00:11.1924744Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpevx_1rs_/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1924997Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2thepc8s 2023-01-11T23:00:11.1925264Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2thepc8s/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1925496Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc9pis0j8 2023-01-11T23:00:11.1925756Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc9pis0j8/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1925982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1926202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1926425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.1926650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.1926794Z fi_getinfo: -61 2023-01-11T23:00:11.1926985Z fi_getinfo: -61 2023-01-11T23:00:11.1927104Z fi_getinfo: -61 2023-01-11T23:00:11.1927241Z fi_getinfo: -61 2023-01-11T23:00:11.1927344Z ok (10.415s) 2023-01-11T23:00:11.1927363Z 2023-01-11T23:00:11.1927622Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1927737Z Ran 1 test in 10.415s 2023-01-11T23:00:11.1927757Z 2023-01-11T23:00:11.1927852Z OK 2023-01-11T23:00:11.1927871Z 2023-01-11T23:00:11.1927995Z Generating XML reports... 2023-01-11T23:00:11.1928544Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224756.xml 2023-01-11T23:00:11.1928892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1929068Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1929441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1929636Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1929886Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3b8v_0ny 2023-01-11T23:00:11.1930150Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3b8v_0ny/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1930169Z 2023-01-11T23:00:11.1930275Z Running tests... 2023-01-11T23:00:11.1930794Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1931161Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1931448Z test_device_map_gpu_mixed_self_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1931666Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90922 2023-01-11T23:00:11.1931882Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90923 2023-01-11T23:00:11.1932102Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 90924 2023-01-11T23:00:11.1932308Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 90925 2023-01-11T23:00:11.1932686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1932858Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1933237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1933426Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1933772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1933944Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1934320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1934507Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1934865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1935034Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1935411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1935600Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1935945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1936116Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1936484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1936734Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1936986Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpso71qssk 2023-01-11T23:00:11.1937257Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpso71qssk/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1937510Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqmt0y7wb 2023-01-11T23:00:11.1937774Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqmt0y7wb/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1938025Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpprhq8sbh 2023-01-11T23:00:11.1938269Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpprhq8sbh/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1938515Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbc944od5 2023-01-11T23:00:11.1938780Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbc944od5/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1939005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1939229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.1939455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.1939718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1939870Z fi_getinfo: -61 2023-01-11T23:00:11.1939990Z fi_getinfo: -61 2023-01-11T23:00:11.1940130Z fi_getinfo: -61 2023-01-11T23:00:11.1940267Z fi_getinfo: -61 2023-01-11T23:00:11.1940366Z ok (10.424s) 2023-01-11T23:00:11.1940386Z 2023-01-11T23:00:11.1940647Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1940763Z Ran 1 test in 10.425s 2023-01-11T23:00:11.1940785Z 2023-01-11T23:00:11.1940878Z OK 2023-01-11T23:00:11.1940897Z 2023-01-11T23:00:11.1941023Z Generating XML reports... 2023-01-11T23:00:11.1941552Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224809.xml 2023-01-11T23:00:11.1941922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1942103Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1942480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1942668Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1942923Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6i_qkuhi 2023-01-11T23:00:11.1943186Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6i_qkuhi/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1943208Z 2023-01-11T23:00:11.1943317Z Running tests... 2023-01-11T23:00:11.1943563Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1943922Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1944227Z test_device_map_gpu_mixed_self_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1944445Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91273 2023-01-11T23:00:11.1944661Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91274 2023-01-11T23:00:11.1944871Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 91275 2023-01-11T23:00:11.1945081Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 91276 2023-01-11T23:00:11.1945447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1945677Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1946034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1946220Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1946581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1946755Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1947124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1947310Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1947670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1947844Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1948212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1948380Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1948744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1948961Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1949331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1949517Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1949771Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps78kezgt 2023-01-11T23:00:11.1950043Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps78kezgt/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1950300Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj5yny5o7 2023-01-11T23:00:11.1950545Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj5yny5o7/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1950794Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg9iicy2s 2023-01-11T23:00:11.1951060Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg9iicy2s/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1951307Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7tz70og9 2023-01-11T23:00:11.1951572Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7tz70og9/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1951798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.1952024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1952251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1952474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.1952602Z fi_getinfo: -61 2023-01-11T23:00:11.1952744Z fi_getinfo: -61 2023-01-11T23:00:11.1952883Z fi_getinfo: -61 2023-01-11T23:00:11.1953018Z fi_getinfo: -61 2023-01-11T23:00:11.1953121Z ok (10.280s) 2023-01-11T23:00:11.1953144Z 2023-01-11T23:00:11.1953412Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1953530Z Ran 1 test in 10.280s 2023-01-11T23:00:11.1953549Z 2023-01-11T23:00:11.1953624Z OK 2023-01-11T23:00:11.1953661Z 2023-01-11T23:00:11.1953767Z Generating XML reports... 2023-01-11T23:00:11.1954313Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224822.xml 2023-01-11T23:00:11.1954804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1954976Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1955353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1955544Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1955801Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl131_x7g 2023-01-11T23:00:11.1956061Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl131_x7g/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1956080Z 2023-01-11T23:00:11.1956170Z Running tests... 2023-01-11T23:00:11.1956434Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1956791Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1957102Z test_device_map_gpu_mixed_self_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1957315Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91624 2023-01-11T23:00:11.1957531Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91625 2023-01-11T23:00:11.1957748Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 91626 2023-01-11T23:00:11.1958004Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 91627 2023-01-11T23:00:11.1958384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1958540Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1958914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1959109Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1959471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1959642Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1960013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1960205Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1960560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1960710Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1961077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1961261Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1961627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1961792Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1962156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1962345Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1962600Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmfe163c3 2023-01-11T23:00:11.1962865Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmfe163c3/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1963098Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjjerwq96 2023-01-11T23:00:11.1963366Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjjerwq96/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1963670Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpftdq0tk2 2023-01-11T23:00:11.1963932Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpftdq0tk2/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1964179Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpueu8869m 2023-01-11T23:00:11.1964445Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpueu8869m/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1964673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1964897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.1965124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1965329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.1965479Z fi_getinfo: -61 2023-01-11T23:00:11.1965622Z fi_getinfo: -61 2023-01-11T23:00:11.1965759Z fi_getinfo: -61 2023-01-11T23:00:11.1965896Z fi_getinfo: -61 2023-01-11T23:00:11.1965996Z ok (10.401s) 2023-01-11T23:00:11.1966015Z 2023-01-11T23:00:11.1966282Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1966378Z Ran 1 test in 10.401s 2023-01-11T23:00:11.1966397Z 2023-01-11T23:00:11.1966490Z OK 2023-01-11T23:00:11.1966509Z 2023-01-11T23:00:11.1966632Z Generating XML reports... 2023-01-11T23:00:11.1967221Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224835.xml 2023-01-11T23:00:11.1967598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1967769Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1968141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1968335Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1968628Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv1dx_jes 2023-01-11T23:00:11.1968874Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv1dx_jes/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1968893Z 2023-01-11T23:00:11.1969007Z Running tests... 2023-01-11T23:00:11.1969274Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1969637Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1969940Z test_device_map_gpu_mixed_self_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1970154Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91975 2023-01-11T23:00:11.1970689Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91976 2023-01-11T23:00:11.1970917Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 91977 2023-01-11T23:00:11.1971106Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 91978 2023-01-11T23:00:11.1971488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1971662Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1972040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1972225Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1972583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1972751Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1973221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1973405Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1973745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1973917Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1974288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1974471Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1974837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1975005Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1975369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1975561Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1975811Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppi4uw05i 2023-01-11T23:00:11.1976058Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppi4uw05i/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1976371Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphh4ea3ld 2023-01-11T23:00:11.1976645Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphh4ea3ld/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1976890Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5v9yfa3g 2023-01-11T23:00:11.1977156Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5v9yfa3g/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1977401Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_di83wg3 2023-01-11T23:00:11.1977661Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_di83wg3/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1977890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1978096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.1978321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.1978546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1978696Z fi_getinfo: -61 2023-01-11T23:00:11.1978833Z fi_getinfo: -61 2023-01-11T23:00:11.1978969Z fi_getinfo: -61 2023-01-11T23:00:11.1979102Z fi_getinfo: -61 2023-01-11T23:00:11.1979184Z ok (10.370s) 2023-01-11T23:00:11.1979220Z 2023-01-11T23:00:11.1979468Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1979582Z Ran 1 test in 10.371s 2023-01-11T23:00:11.1979601Z 2023-01-11T23:00:11.1979698Z OK 2023-01-11T23:00:11.1979716Z 2023-01-11T23:00:11.1979837Z Generating XML reports... 2023-01-11T23:00:11.1980381Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224849.xml 2023-01-11T23:00:11.1980750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1980931Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1981302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1981472Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1981726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp18huzfap 2023-01-11T23:00:11.1981991Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp18huzfap/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1982060Z 2023-01-11T23:00:11.1982175Z Running tests... 2023-01-11T23:00:11.1982438Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1982795Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1983100Z test_device_map_gpu_mixed_self_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1983313Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92326 2023-01-11T23:00:11.1983524Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92327 2023-01-11T23:00:11.1983717Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 92328 2023-01-11T23:00:11.1983922Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 92329 2023-01-11T23:00:11.1984289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1984470Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1984843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1985031Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1985451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1985626Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1986002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1986170Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1986528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1986700Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1987076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1987260Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1987623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1987791Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1988159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1988323Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1988580Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvxni65zt 2023-01-11T23:00:11.1988846Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvxni65zt/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1989101Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbthwntc_ 2023-01-11T23:00:11.1989363Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbthwntc_/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1989607Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5967ohgg 2023-01-11T23:00:11.1989871Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5967ohgg/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1990114Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp35ds8e9q 2023-01-11T23:00:11.1990377Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp35ds8e9q/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1990584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.1990860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.1991081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.1991305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.1991450Z fi_getinfo: -61 2023-01-11T23:00:11.1991586Z fi_getinfo: -61 2023-01-11T23:00:11.1991720Z fi_getinfo: -61 2023-01-11T23:00:11.1991860Z fi_getinfo: -61 2023-01-11T23:00:11.1991946Z ok (10.378s) 2023-01-11T23:00:11.1991966Z 2023-01-11T23:00:11.1992225Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1992337Z Ran 1 test in 10.378s 2023-01-11T23:00:11.1992356Z 2023-01-11T23:00:11.1992453Z OK 2023-01-11T23:00:11.1992472Z 2023-01-11T23:00:11.1992593Z Generating XML reports... 2023-01-11T23:00:11.1993132Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224902.xml 2023-01-11T23:00:11.1993505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1993675Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1994031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1994337Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1994594Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpneizl0eg 2023-01-11T23:00:11.1994862Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpneizl0eg/_remote_module_non_scriptable.py 2023-01-11T23:00:11.1994881Z 2023-01-11T23:00:11.1994988Z Running tests... 2023-01-11T23:00:11.1995248Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.1995606Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.1995913Z test_device_map_gpu_mixed_self_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.1996130Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92677 2023-01-11T23:00:11.1996326Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92678 2023-01-11T23:00:11.1996539Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 92679 2023-01-11T23:00:11.1996744Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 92680 2023-01-11T23:00:11.1997111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1997286Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1997659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1997854Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1998214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1998365Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1998731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.1998922Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.1999282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.1999448Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.1999816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2000051Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2000412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2000577Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2000923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2001114Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2001368Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg8op2csc 2023-01-11T23:00:11.2001633Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg8op2csc/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2001884Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7k9_rjun 2023-01-11T23:00:11.2002144Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7k9_rjun/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2002392Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmc6e1trn 2023-01-11T23:00:11.2002657Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmc6e1trn/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2002884Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpebmtlsm7 2023-01-11T23:00:11.2003187Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpebmtlsm7/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2003421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2003642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2003862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2004085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2004233Z fi_getinfo: -61 2023-01-11T23:00:11.2004371Z fi_getinfo: -61 2023-01-11T23:00:11.2004488Z fi_getinfo: -61 2023-01-11T23:00:11.2004629Z fi_getinfo: -61 2023-01-11T23:00:11.2004727Z ok (10.325s) 2023-01-11T23:00:11.2004746Z 2023-01-11T23:00:11.2005006Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2005118Z Ran 1 test in 10.326s 2023-01-11T23:00:11.2005138Z 2023-01-11T23:00:11.2005229Z OK 2023-01-11T23:00:11.2005248Z 2023-01-11T23:00:11.2005372Z Generating XML reports... 2023-01-11T23:00:11.2005920Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224915.xml 2023-01-11T23:00:11.2006267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2006443Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2006814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2007008Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2007255Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3nq24e7o 2023-01-11T23:00:11.2007516Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3nq24e7o/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2007535Z 2023-01-11T23:00:11.2007648Z Running tests... 2023-01-11T23:00:11.2007907Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2008266Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2008552Z test_device_map_gpu_mixed_self_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2008766Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93028 2023-01-11T23:00:11.2009041Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93029 2023-01-11T23:00:11.2009256Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 93030 2023-01-11T23:00:11.2009461Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 93031 2023-01-11T23:00:11.2009831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2010009Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2010592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2010793Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2011142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2011311Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2011689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2011873Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2012229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2012397Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2012826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2013022Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2013368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2013535Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2013897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2014077Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2014328Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjxs4sklk 2023-01-11T23:00:11.2014591Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjxs4sklk/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2014842Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvxkuewj6 2023-01-11T23:00:11.2015106Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvxkuewj6/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2015351Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpddwzkctf 2023-01-11T23:00:11.2015596Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpddwzkctf/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2015845Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprx_zpmlj 2023-01-11T23:00:11.2016099Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprx_zpmlj/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2016322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2016542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2016766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2016985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2017130Z fi_getinfo: -61 2023-01-11T23:00:11.2017254Z fi_getinfo: -61 2023-01-11T23:00:11.2017390Z fi_getinfo: -61 2023-01-11T23:00:11.2017525Z fi_getinfo: -61 2023-01-11T23:00:11.2017623Z ok (10.419s) 2023-01-11T23:00:11.2017642Z 2023-01-11T23:00:11.2017902Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2018084Z Ran 1 test in 10.420s 2023-01-11T23:00:11.2018103Z 2023-01-11T23:00:11.2018195Z OK 2023-01-11T23:00:11.2018214Z 2023-01-11T23:00:11.2018335Z Generating XML reports... 2023-01-11T23:00:11.2018864Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224928.xml 2023-01-11T23:00:11.2019228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2019398Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2019770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2019954Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2020201Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg_3_9_j9 2023-01-11T23:00:11.2020460Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg_3_9_j9/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2020479Z 2023-01-11T23:00:11.2020585Z Running tests... 2023-01-11T23:00:11.2020829Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2021183Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2021534Z test_device_map_gpu_mixed_self_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2021750Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93379 2023-01-11T23:00:11.2021963Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93380 2023-01-11T23:00:11.2022173Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 93381 2023-01-11T23:00:11.2022377Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 93382 2023-01-11T23:00:11.2022750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2022920Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2023277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2023463Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2023827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2023998Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2024365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2024551Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2024906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2025077Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2025426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2025609Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2025970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2026138Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2026501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2026683Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2026934Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9hkuyup8 2023-01-11T23:00:11.2027251Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9hkuyup8/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2027499Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplvabne0s 2023-01-11T23:00:11.2027748Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplvabne0s/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2027999Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp92an8n_q 2023-01-11T23:00:11.2028258Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp92an8n_q/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2028503Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmua5znhi 2023-01-11T23:00:11.2028760Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmua5znhi/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2028983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2029209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2029429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2029633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2029779Z fi_getinfo: -61 2023-01-11T23:00:11.2029913Z fi_getinfo: -61 2023-01-11T23:00:11.2030048Z fi_getinfo: -61 2023-01-11T23:00:11.2030228Z fi_getinfo: -61 2023-01-11T23:00:11.2030331Z ok (10.428s) 2023-01-11T23:00:11.2030351Z 2023-01-11T23:00:11.2030618Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2030715Z Ran 1 test in 10.428s 2023-01-11T23:00:11.2030750Z 2023-01-11T23:00:11.2030825Z OK 2023-01-11T23:00:11.2030843Z 2023-01-11T23:00:11.2030965Z Generating XML reports... 2023-01-11T23:00:11.2031506Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224941.xml 2023-01-11T23:00:11.2031876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2032047Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2032421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2032609Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2032861Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl0srub8a 2023-01-11T23:00:11.2033107Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl0srub8a/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2033147Z 2023-01-11T23:00:11.2033238Z Running tests... 2023-01-11T23:00:11.2033506Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2033860Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2034170Z test_device_map_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2034391Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93730 2023-01-11T23:00:11.2034604Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93731 2023-01-11T23:00:11.2034823Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 93732 2023-01-11T23:00:11.2035035Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 93733 2023-01-11T23:00:11.2035387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2035564Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2035938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2036180Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2036539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2036706Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2037077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2037263Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2037619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2037770Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2038137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2038324Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2038684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2038851Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2039213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2039436Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2039692Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfrdcq6mh 2023-01-11T23:00:11.2039940Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfrdcq6mh/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2040189Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpybaxj00e 2023-01-11T23:00:11.2040452Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpybaxj00e/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2040702Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp797thbxh 2023-01-11T23:00:11.2040961Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp797thbxh/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2041204Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_tl4__ov 2023-01-11T23:00:11.2041459Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_tl4__ov/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2041683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2041908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2042112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2042333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2042483Z fi_getinfo: -61 2023-01-11T23:00:11.2042619Z fi_getinfo: -61 2023-01-11T23:00:11.2042754Z fi_getinfo: -61 2023-01-11T23:00:11.2042889Z fi_getinfo: -61 2023-01-11T23:00:11.2042987Z ok (7.910s) 2023-01-11T23:00:11.2043007Z 2023-01-11T23:00:11.2043252Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2043362Z Ran 1 test in 7.911s 2023-01-11T23:00:11.2043381Z 2023-01-11T23:00:11.2043473Z OK 2023-01-11T23:00:11.2043496Z 2023-01-11T23:00:11.2043618Z Generating XML reports... 2023-01-11T23:00:11.2044162Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224954.xml 2023-01-11T23:00:11.2044527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2044699Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2045132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2045320Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2045554Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgpwn1eui 2023-01-11T23:00:11.2045817Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgpwn1eui/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2045836Z 2023-01-11T23:00:11.2045946Z Running tests... 2023-01-11T23:00:11.2046207Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2046560Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2046876Z test_device_map_gpu_non_default_to_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2047091Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94077 2023-01-11T23:00:11.2047308Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94078 2023-01-11T23:00:11.2047501Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 94079 2023-01-11T23:00:11.2047709Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 94080 2023-01-11T23:00:11.2048076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2048304Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2048685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2048873Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2049231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2049402Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2049767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2049937Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2050487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2050678Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2051047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2051230Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2051592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2051760Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2052125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2052290Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2052543Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkawenr_n 2023-01-11T23:00:11.2052813Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkawenr_n/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2053063Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm9w1kidh 2023-01-11T23:00:11.2053326Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm9w1kidh/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2053573Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqc61lp3v 2023-01-11T23:00:11.2053818Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1jbxms5g 2023-01-11T23:00:11.2054156Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqc61lp3v/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2054411Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1jbxms5g/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2054618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2054876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2055103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2055323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2055469Z fi_getinfo: -61 2023-01-11T23:00:11.2055606Z fi_getinfo: -61 2023-01-11T23:00:11.2055742Z fi_getinfo: -61 2023-01-11T23:00:11.2055861Z fi_getinfo: -61 2023-01-11T23:00:11.2055959Z ok (10.477s) 2023-01-11T23:00:11.2055979Z 2023-01-11T23:00:11.2056245Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2056358Z Ran 1 test in 10.477s 2023-01-11T23:00:11.2056378Z 2023-01-11T23:00:11.2056470Z OK 2023-01-11T23:00:11.2056489Z 2023-01-11T23:00:11.2056609Z Generating XML reports... 2023-01-11T23:00:11.2057150Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225005.xml 2023-01-11T23:00:11.2057577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2057759Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2058121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2058310Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2058560Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi_uxsz3l 2023-01-11T23:00:11.2058827Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi_uxsz3l/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2058847Z 2023-01-11T23:00:11.2058953Z Running tests... 2023-01-11T23:00:11.2059214Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2059569Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2059881Z test_device_map_gpu_to_cpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2060082Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94436 2023-01-11T23:00:11.2060295Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94437 2023-01-11T23:00:11.2060504Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 94438 2023-01-11T23:00:11.2060710Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 94439 2023-01-11T23:00:11.2061082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2061253Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2061627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2061816Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2062174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2062327Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2062694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2062877Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2063287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2063454Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2063819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2064004Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2064360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2064511Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2064875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2065056Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2065312Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplo99h9um 2023-01-11T23:00:11.2065578Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplo99h9um/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2065831Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprbabpwc_ 2023-01-11T23:00:11.2066096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprbabpwc_/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2066389Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiupoh9m9 2023-01-11T23:00:11.2066654Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiupoh9m9/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2066883Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfcnl8i93 2023-01-11T23:00:11.2067142Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfcnl8i93/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2067367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2067596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2067813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2068031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2068179Z fi_getinfo: -61 2023-01-11T23:00:11.2068315Z fi_getinfo: -61 2023-01-11T23:00:11.2068438Z fi_getinfo: -61 2023-01-11T23:00:11.2070039Z fi_getinfo: -61 2023-01-11T23:00:11.2070140Z ok (7.911s) 2023-01-11T23:00:11.2070159Z 2023-01-11T23:00:11.2070419Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2070531Z Ran 1 test in 7.911s 2023-01-11T23:00:11.2070550Z 2023-01-11T23:00:11.2070640Z OK 2023-01-11T23:00:11.2070660Z 2023-01-11T23:00:11.2070781Z Generating XML reports... 2023-01-11T23:00:11.2071321Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225018.xml 2023-01-11T23:00:11.2071673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2071846Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2072219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2072409Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2072658Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1gz4m1_9 2023-01-11T23:00:11.2072921Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1gz4m1_9/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2072940Z 2023-01-11T23:00:11.2073047Z Running tests... 2023-01-11T23:00:11.2073308Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2073710Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2074024Z test_device_map_gpu_to_cpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2074238Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94787 2023-01-11T23:00:11.2074448Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94788 2023-01-11T23:00:11.2074662Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 94789 2023-01-11T23:00:11.2074871Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 94790 2023-01-11T23:00:11.2075236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2075407Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2075782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2075956Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2076321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2076489Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2076910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2077103Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2077463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2077633Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2077998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2078184Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2078522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2078691Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2079058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2079239Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2079488Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe6v240xy 2023-01-11T23:00:11.2079749Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe6v240xy/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2079995Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa5rl0l0b 2023-01-11T23:00:11.2080259Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa5rl0l0b/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2080488Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp08uametv 2023-01-11T23:00:11.2080733Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpch0m5gan 2023-01-11T23:00:11.2080990Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp08uametv/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2081246Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpch0m5gan/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2081471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2081695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2081916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2082189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2082318Z fi_getinfo: -61 2023-01-11T23:00:11.2082454Z fi_getinfo: -61 2023-01-11T23:00:11.2082588Z fi_getinfo: -61 2023-01-11T23:00:11.2082723Z fi_getinfo: -61 2023-01-11T23:00:11.2082821Z ok (7.908s) 2023-01-11T23:00:11.2082840Z 2023-01-11T23:00:11.2083099Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2083210Z Ran 1 test in 7.908s 2023-01-11T23:00:11.2083229Z 2023-01-11T23:00:11.2083324Z OK 2023-01-11T23:00:11.2083343Z 2023-01-11T23:00:11.2083448Z Generating XML reports... 2023-01-11T23:00:11.2083989Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225029.xml 2023-01-11T23:00:11.2084352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2084523Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2084900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2085085Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2085333Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplo9evtbv 2023-01-11T23:00:11.2085646Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplo9evtbv/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2085667Z 2023-01-11T23:00:11.2085778Z Running tests... 2023-01-11T23:00:11.2086023Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2086374Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2086665Z test_device_maps_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2086880Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95138 2023-01-11T23:00:11.2087095Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95139 2023-01-11T23:00:11.2087303Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 95140 2023-01-11T23:00:11.2087512Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 95141 2023-01-11T23:00:11.2087880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2088034Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2088411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2088598Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2088954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2089125Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2089491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2089675Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2090028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2090182Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2090851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2091039Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2091402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2091644Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2111613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2111838Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2112100Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg9tiyw45 2023-01-11T23:00:11.2112376Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg9tiyw45/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2112628Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph_w0s9hd 2023-01-11T23:00:11.2112889Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph_w0s9hd/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2113138Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpos9xd8ku 2023-01-11T23:00:11.2113392Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpos9xd8ku/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2113627Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu6p61vcd 2023-01-11T23:00:11.2113876Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu6p61vcd/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2114090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2114438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2114669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2114879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2115052Z fi_getinfo: -61 2023-01-11T23:00:11.2115180Z fi_getinfo: -61 2023-01-11T23:00:11.2115299Z fi_getinfo: -61 2023-01-11T23:00:11.2115424Z fi_getinfo: -61 2023-01-11T23:00:11.2115511Z ok (10.434s) 2023-01-11T23:00:11.2115537Z 2023-01-11T23:00:11.2115796Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2115900Z Ran 1 test in 10.434s 2023-01-11T23:00:11.2115919Z 2023-01-11T23:00:11.2116011Z OK 2023-01-11T23:00:11.2116030Z 2023-01-11T23:00:11.2116150Z Generating XML reports... 2023-01-11T23:00:11.2116678Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225040.xml 2023-01-11T23:00:11.2117051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2117221Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2117598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2117785Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2118036Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4_33nf9c 2023-01-11T23:00:11.2118297Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4_33nf9c/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2118317Z 2023-01-11T23:00:11.2118423Z Running tests... 2023-01-11T23:00:11.2118681Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2119019Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2119317Z test_device_maps_in_options (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2119530Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95497 2023-01-11T23:00:11.2119737Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95498 2023-01-11T23:00:11.2119939Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 95499 2023-01-11T23:00:11.2120139Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 95500 2023-01-11T23:00:11.2120571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2120738Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2121098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2121282Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2121634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2121799Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2122168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2122354Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2122712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2122873Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2123226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2123393Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2123790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2123954Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2124310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2124483Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2124728Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5osubakq 2023-01-11T23:00:11.2124985Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5osubakq/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2125224Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4q3_duow 2023-01-11T23:00:11.2125477Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4q3_duow/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2125711Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp24ulcesl 2023-01-11T23:00:11.2125964Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp24ulcesl/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2126199Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5k9vpnnw 2023-01-11T23:00:11.2126448Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5k9vpnnw/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2126661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2126877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2127088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2127297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2127428Z fi_getinfo: -61 2023-01-11T23:00:11.2127556Z fi_getinfo: -61 2023-01-11T23:00:11.2127686Z fi_getinfo: -61 2023-01-11T23:00:11.2127812Z fi_getinfo: -61 2023-01-11T23:00:11.2127900Z ok (10.518s) 2023-01-11T23:00:11.2127919Z 2023-01-11T23:00:11.2128173Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2128275Z Ran 1 test in 10.518s 2023-01-11T23:00:11.2128294Z 2023-01-11T23:00:11.2128369Z OK 2023-01-11T23:00:11.2128394Z 2023-01-11T23:00:11.2128500Z Generating XML reports... 2023-01-11T23:00:11.2129031Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225053.xml 2023-01-11T23:00:11.2129446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2129607Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2129974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2130152Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2130732Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwcyq_bkd 2023-01-11T23:00:11.2131000Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwcyq_bkd/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2131021Z 2023-01-11T23:00:11.2131112Z Running tests... 2023-01-11T23:00:11.2131374Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2131725Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2132034Z test_device_maps_invalid_max_local_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2132240Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95856 2023-01-11T23:00:11.2132537Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95857 2023-01-11T23:00:11.2132748Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 95858 2023-01-11T23:00:11.2132944Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 95859 2023-01-11T23:00:11.2133301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2133464Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2133834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2134011Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2134362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2134522Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2134882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2135058Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2135403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2135555Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2135906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2136084Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2136435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2136594Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2136950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2137124Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2137366Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxvheeo5q 2023-01-11T23:00:11.2137619Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxvheeo5q/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2137859Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjgtp8_l3 2023-01-11T23:00:11.2138181Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjgtp8_l3/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2138420Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppcrguu_m 2023-01-11T23:00:11.2138686Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppcrguu_m/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2138934Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvptp08k9 2023-01-11T23:00:11.2139186Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvptp08k9/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2139405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2139618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2139825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2140041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2140186Z fi_getinfo: -61 2023-01-11T23:00:11.2140321Z fi_getinfo: -61 2023-01-11T23:00:11.2140455Z fi_getinfo: -61 2023-01-11T23:00:11.2140586Z fi_getinfo: -61 2023-01-11T23:00:11.2140684Z ok (4.602s) 2023-01-11T23:00:11.2140704Z 2023-01-11T23:00:11.2140951Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2141063Z Ran 1 test in 4.602s 2023-01-11T23:00:11.2141130Z 2023-01-11T23:00:11.2141227Z OK 2023-01-11T23:00:11.2141246Z 2023-01-11T23:00:11.2141370Z Generating XML reports... 2023-01-11T23:00:11.2141913Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225106.xml 2023-01-11T23:00:11.2142275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2142450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2142820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2143007Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2143240Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9orhxs0w 2023-01-11T23:00:11.2143504Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9orhxs0w/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2143523Z 2023-01-11T23:00:11.2143630Z Running tests... 2023-01-11T23:00:11.2143887Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2144238Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2144554Z test_device_maps_invalid_max_remote_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2144772Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96043 2023-01-11T23:00:11.2144985Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96044 2023-01-11T23:00:11.2145178Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 96045 2023-01-11T23:00:11.2145385Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 96046 2023-01-11T23:00:11.2145756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2145925Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2146300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2146486Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2146844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2147074Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2147441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2147610Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2147965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2148134Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2148501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2148682Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2149040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2149211Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2149575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2149756Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2149991Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp34qtn_ud 2023-01-11T23:00:11.2150320Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp34qtn_ud/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2150577Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxqbj3bej 2023-01-11T23:00:11.2150842Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxqbj3bej/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2151087Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6_0j8q7z 2023-01-11T23:00:11.2151346Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6_0j8q7z/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2151595Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp69u9y3_s 2023-01-11T23:00:11.2151851Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp69u9y3_s/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2152059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2152284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2152503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2152725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2152871Z fi_getinfo: -61 2023-01-11T23:00:11.2153008Z fi_getinfo: -61 2023-01-11T23:00:11.2153142Z fi_getinfo: -61 2023-01-11T23:00:11.2153277Z fi_getinfo: -61 2023-01-11T23:00:11.2153364Z ok (4.603s) 2023-01-11T23:00:11.2153384Z 2023-01-11T23:00:11.2153644Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2153756Z Ran 1 test in 4.603s 2023-01-11T23:00:11.2153775Z 2023-01-11T23:00:11.2153866Z OK 2023-01-11T23:00:11.2153885Z 2023-01-11T23:00:11.2154005Z Generating XML reports... 2023-01-11T23:00:11.2154551Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225113.xml 2023-01-11T23:00:11.2154962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2155134Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2155492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2155680Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2156019Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm70av3fg 2023-01-11T23:00:11.2156284Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm70av3fg/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2156304Z 2023-01-11T23:00:11.2156411Z Running tests... 2023-01-11T23:00:11.2156672Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2157030Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2157340Z test_device_maps_invalid_min_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2157553Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96230 2023-01-11T23:00:11.2157748Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96231 2023-01-11T23:00:11.2157956Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 96232 2023-01-11T23:00:11.2158165Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 96233 2023-01-11T23:00:11.2158534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2158704Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2159077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2159301Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2159672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2159823Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2160191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2160384Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2160744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2160911Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2161275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2161462Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2161822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2161972Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2162336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2162519Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2162774Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpilx_fng7 2023-01-11T23:00:11.2163040Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpilx_fng7/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2163292Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsa5ud1pv 2023-01-11T23:00:11.2163557Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsa5ud1pv/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2163804Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp88uas1e8 2023-01-11T23:00:11.2164062Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp88uas1e8/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2164290Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ujpevc0 2023-01-11T23:00:11.2164550Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ujpevc0/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2164821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2165043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2165264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2165484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2165632Z fi_getinfo: -61 2023-01-11T23:00:11.2165770Z fi_getinfo: -61 2023-01-11T23:00:11.2165888Z fi_getinfo: -61 2023-01-11T23:00:11.2166022Z fi_getinfo: -61 2023-01-11T23:00:11.2166121Z ok (4.565s) 2023-01-11T23:00:11.2166141Z 2023-01-11T23:00:11.2166402Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2166515Z Ran 1 test in 4.565s 2023-01-11T23:00:11.2166535Z 2023-01-11T23:00:11.2166626Z OK 2023-01-11T23:00:11.2166645Z 2023-01-11T23:00:11.2166766Z Generating XML reports... 2023-01-11T23:00:11.2167313Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225121.xml 2023-01-11T23:00:11.2167664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2167836Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2168249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2168441Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2168690Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxyiyc38v 2023-01-11T23:00:11.2168952Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxyiyc38v/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2168972Z 2023-01-11T23:00:11.2169078Z Running tests... 2023-01-11T23:00:11.2169344Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2169678Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2169976Z test_device_maps_many_to_one (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2170190Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96405 2023-01-11T23:00:11.2170634Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96406 2023-01-11T23:00:11.2170852Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 96407 2023-01-11T23:00:11.2171060Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 96408 2023-01-11T23:00:11.2171431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2171603Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2171982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2172152Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2172506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2172675Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2173046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2173232Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2173588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2173757Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2174213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2174379Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2174741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2174908Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2175273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2175457Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2175712Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoxm8o0h9 2023-01-11T23:00:11.2175975Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoxm8o0h9/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2176224Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmgjcg_23 2023-01-11T23:00:11.2176487Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmgjcg_23/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2176717Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3jwi0mou 2023-01-11T23:00:11.2176976Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3jwi0mou/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2177276Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7ojc2g3u 2023-01-11T23:00:11.2177542Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7ojc2g3u/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2177767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2177988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2178209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2178432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2178564Z fi_getinfo: -61 2023-01-11T23:00:11.2178702Z fi_getinfo: -61 2023-01-11T23:00:11.2178837Z fi_getinfo: -61 2023-01-11T23:00:11.2178972Z fi_getinfo: -61 2023-01-11T23:00:11.2179070Z ok (4.609s) 2023-01-11T23:00:11.2179090Z 2023-01-11T23:00:11.2179350Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2179464Z Ran 1 test in 4.610s 2023-01-11T23:00:11.2179484Z 2023-01-11T23:00:11.2179573Z OK 2023-01-11T23:00:11.2179593Z 2023-01-11T23:00:11.2179700Z Generating XML reports... 2023-01-11T23:00:11.2180243Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225128.xml 2023-01-11T23:00:11.2180608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2180782Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2181155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2181340Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2181590Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptnhrm3oi 2023-01-11T23:00:11.2181855Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptnhrm3oi/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2181875Z 2023-01-11T23:00:11.2181981Z Running tests... 2023-01-11T23:00:11.2182228Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2182583Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2182886Z test_device_maps_missing_config (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2183151Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96592 2023-01-11T23:00:11.2183364Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96593 2023-01-11T23:00:11.2183574Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 96594 2023-01-11T23:00:11.2183778Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 96595 2023-01-11T23:00:11.2184151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2184308Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2184686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2184873Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2185232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2185404Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2185772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2185960Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2186351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2186509Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2186874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2187058Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2187416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2187587Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2187948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2188131Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2188384Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdwqaqeyf 2023-01-11T23:00:11.2188654Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdwqaqeyf/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2188885Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyblu3rqu 2023-01-11T23:00:11.2189145Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyblu3rqu/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2189391Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgris6t_b 2023-01-11T23:00:11.2189655Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgris6t_b/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2189897Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp26_2af2i 2023-01-11T23:00:11.2190151Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp26_2af2i/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2190375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2190600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2190822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2191026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2191173Z fi_getinfo: -61 2023-01-11T23:00:11.2191308Z fi_getinfo: -61 2023-01-11T23:00:11.2191442Z fi_getinfo: -61 2023-01-11T23:00:11.2191578Z fi_getinfo: -61 2023-01-11T23:00:11.2191722Z ok (6.704s) 2023-01-11T23:00:11.2191741Z 2023-01-11T23:00:11.2192002Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2192098Z Ran 1 test in 6.705s 2023-01-11T23:00:11.2192117Z 2023-01-11T23:00:11.2192209Z OK 2023-01-11T23:00:11.2192228Z 2023-01-11T23:00:11.2192350Z Generating XML reports... 2023-01-11T23:00:11.2192893Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225135.xml 2023-01-11T23:00:11.2193256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2193429Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2193802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2193986Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2194225Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpto97wr_y 2023-01-11T23:00:11.2194487Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpto97wr_y/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2194506Z 2023-01-11T23:00:11.2194613Z Running tests... 2023-01-11T23:00:11.2194875Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2195279Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2195594Z test_device_maps_missing_config_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2195807Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96939 2023-01-11T23:00:11.2196016Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96940 2023-01-11T23:00:11.2196228Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 96941 2023-01-11T23:00:11.2196421Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 96942 2023-01-11T23:00:11.2196790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2196961Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2197335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2197521Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2197875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2198047Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2198415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2198588Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2198943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2199111Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2199475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2199660Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2200021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2200190Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2200552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2200735Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2201018Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiy6bargo 2023-01-11T23:00:11.2201285Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiy6bargo/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2201535Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd5x3_ngx 2023-01-11T23:00:11.2201798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd5x3_ngx/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2202044Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_zdzv_28 2023-01-11T23:00:11.2202301Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_zdzv_28/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2202546Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4wmnax4v 2023-01-11T23:00:11.2202800Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4wmnax4v/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2203028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2203235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2203455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2203675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2203864Z fi_getinfo: -61 2023-01-11T23:00:11.2204005Z fi_getinfo: -61 2023-01-11T23:00:11.2204139Z fi_getinfo: -61 2023-01-11T23:00:11.2204273Z fi_getinfo: -61 2023-01-11T23:00:11.2204356Z ok (6.814s) 2023-01-11T23:00:11.2204375Z 2023-01-11T23:00:11.2204635Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2204746Z Ran 1 test in 6.814s 2023-01-11T23:00:11.2204765Z 2023-01-11T23:00:11.2204857Z OK 2023-01-11T23:00:11.2204876Z 2023-01-11T23:00:11.2205000Z Generating XML reports... 2023-01-11T23:00:11.2205538Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225145.xml 2023-01-11T23:00:11.2205903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2206074Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2206433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2206621Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2206871Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuu127hvo 2023-01-11T23:00:11.2207132Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuu127hvo/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2207151Z 2023-01-11T23:00:11.2207260Z Running tests... 2023-01-11T23:00:11.2207520Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2207873Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2208188Z test_device_maps_missing_config_not_timeout (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2208404Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97286 2023-01-11T23:00:11.2208601Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97287 2023-01-11T23:00:11.2208809Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 97288 2023-01-11T23:00:11.2209013Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 97289 2023-01-11T23:00:11.2209379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2209599Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2209972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2210160Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2210739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2210899Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2211274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2211459Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2211810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2211978Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2212344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2212528Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2212890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2213058Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2213478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2213671Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2213923Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwz8bsbkd 2023-01-11T23:00:11.2214189Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwz8bsbkd/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2214454Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzfx34jno 2023-01-11T23:00:11.2214715Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzfx34jno/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2214964Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiliby5p0 2023-01-11T23:00:11.2215228Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiliby5p0/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2215476Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp49cb7x59 2023-01-11T23:00:11.2215717Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp49cb7x59/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2215957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2216185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2216415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2216650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2216801Z fi_getinfo: -61 2023-01-11T23:00:11.2216938Z fi_getinfo: -61 2023-01-11T23:00:11.2217075Z fi_getinfo: -61 2023-01-11T23:00:11.2217196Z fi_getinfo: -61 2023-01-11T23:00:11.2217296Z ok (6.663s) 2023-01-11T23:00:11.2217318Z 2023-01-11T23:00:11.2217584Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2217698Z Ran 1 test in 6.664s 2023-01-11T23:00:11.2217717Z 2023-01-11T23:00:11.2217808Z OK 2023-01-11T23:00:11.2217827Z 2023-01-11T23:00:11.2217959Z Generating XML reports... 2023-01-11T23:00:11.2218514Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225154.xml 2023-01-11T23:00:11.2218882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2219126Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2219494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2219684Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2219938Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnegi5v6c 2023-01-11T23:00:11.2220211Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnegi5v6c/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2220231Z 2023-01-11T23:00:11.2220338Z Running tests... 2023-01-11T23:00:11.2220597Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2220957Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2221277Z test_device_maps_missing_config_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2221482Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97633 2023-01-11T23:00:11.2221699Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97634 2023-01-11T23:00:11.2221913Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 97635 2023-01-11T23:00:11.2222171Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 97636 2023-01-11T23:00:11.2222555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2222728Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2223108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2223298Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2223666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2223819Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2224195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2224384Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2224750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2224921Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2225291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2225475Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2225845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2226002Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2226371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2226555Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2226815Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp73xvmwo 2023-01-11T23:00:11.2227084Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp73xvmwo/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2227341Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplm5kach2 2023-01-11T23:00:11.2227608Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplm5kach2/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2227861Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq7mqzpty 2023-01-11T23:00:11.2228178Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq7mqzpty/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2228410Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp75c72yqy 2023-01-11T23:00:11.2228670Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp75c72yqy/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2228900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2229129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2229353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2229578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2229723Z fi_getinfo: -61 2023-01-11T23:00:11.2229859Z fi_getinfo: -61 2023-01-11T23:00:11.2229981Z fi_getinfo: -61 2023-01-11T23:00:11.2230116Z fi_getinfo: -61 2023-01-11T23:00:11.2230215Z ok (6.607s) 2023-01-11T23:00:11.2230234Z 2023-01-11T23:00:11.2230497Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2230609Z Ran 1 test in 6.607s 2023-01-11T23:00:11.2230628Z 2023-01-11T23:00:11.2230719Z OK 2023-01-11T23:00:11.2230738Z 2023-01-11T23:00:11.2230860Z Generating XML reports... 2023-01-11T23:00:11.2231455Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225204.xml 2023-01-11T23:00:11.2231820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2231995Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2232376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2232570Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2232827Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpllep6zoh 2023-01-11T23:00:11.2233096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpllep6zoh/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2233115Z 2023-01-11T23:00:11.2233223Z Running tests... 2023-01-11T23:00:11.2233483Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2233827Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2234164Z test_device_maps_missing_config_remote_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2234382Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97980 2023-01-11T23:00:11.2234597Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97981 2023-01-11T23:00:11.2234814Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 97982 2023-01-11T23:00:11.2235024Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 97983 2023-01-11T23:00:11.2235399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2235572Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2235956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2236132Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2236496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2236667Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2237042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2237282Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2237647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2237819Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2238194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2238365Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2238733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2238902Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2239271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2239460Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2239716Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo36_nsi4 2023-01-11T23:00:11.2239983Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo36_nsi4/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2240238Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq445hab8 2023-01-11T23:00:11.2240561Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq445hab8/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2240804Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwiio56pl 2023-01-11T23:00:11.2241072Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwiio56pl/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2241329Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcbopwgh_ 2023-01-11T23:00:11.2241597Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcbopwgh_/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2241827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2242056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2242281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2242510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2242659Z fi_getinfo: -61 2023-01-11T23:00:11.2242778Z fi_getinfo: -61 2023-01-11T23:00:11.2242915Z fi_getinfo: -61 2023-01-11T23:00:11.2243050Z fi_getinfo: -61 2023-01-11T23:00:11.2243149Z ok (6.583s) 2023-01-11T23:00:11.2243169Z 2023-01-11T23:00:11.2243432Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2243545Z Ran 1 test in 6.583s 2023-01-11T23:00:11.2243564Z 2023-01-11T23:00:11.2243659Z OK 2023-01-11T23:00:11.2243678Z 2023-01-11T23:00:11.2243784Z Generating XML reports... 2023-01-11T23:00:11.2244334Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225213.xml 2023-01-11T23:00:11.2244706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2244884Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2245267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2245457Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2245713Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwnfpt7kt 2023-01-11T23:00:11.2245984Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwnfpt7kt/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2246054Z 2023-01-11T23:00:11.2246164Z Running tests... 2023-01-11T23:00:11.2246414Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2246773Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2247100Z test_device_maps_missing_config_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2247322Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98327 2023-01-11T23:00:11.2247536Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98328 2023-01-11T23:00:11.2247750Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 98329 2023-01-11T23:00:11.2247961Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 98330 2023-01-11T23:00:11.2248336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2248514Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2248879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2249069Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2249479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2249656Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2250030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2250221Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2250808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2250985Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2251340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2251526Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2251894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2252067Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2252434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2252619Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2252874Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw4mz7_uy 2023-01-11T23:00:11.2253142Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw4mz7_uy/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2253400Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbf0bny8i 2023-01-11T23:00:11.2253652Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbf0bny8i/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2253905Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpns55xuaz 2023-01-11T23:00:11.2254173Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpns55xuaz/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2254423Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzz3odrz5 2023-01-11T23:00:11.2254686Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzz3odrz5/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2254953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2255183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2255496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2255704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2255852Z fi_getinfo: -61 2023-01-11T23:00:11.2255989Z fi_getinfo: -61 2023-01-11T23:00:11.2256122Z fi_getinfo: -61 2023-01-11T23:00:11.2256256Z fi_getinfo: -61 2023-01-11T23:00:11.2256354Z ok (6.690s) 2023-01-11T23:00:11.2256374Z 2023-01-11T23:00:11.2256641Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2256738Z Ran 1 test in 6.690s 2023-01-11T23:00:11.2256773Z 2023-01-11T23:00:11.2256848Z OK 2023-01-11T23:00:11.2256867Z 2023-01-11T23:00:11.2256989Z Generating XML reports... 2023-01-11T23:00:11.2257538Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225223.xml 2023-01-11T23:00:11.2257912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2258085Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2258464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2258656Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2258983Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpooz7ch90 2023-01-11T23:00:11.2259243Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpooz7ch90/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2259279Z 2023-01-11T23:00:11.2259369Z Running tests... 2023-01-11T23:00:11.2259632Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2259988Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2260325Z test_device_maps_missing_config_response_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2260544Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98674 2023-01-11T23:00:11.2260761Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98675 2023-01-11T23:00:11.2260975Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 98676 2023-01-11T23:00:11.2261190Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 98677 2023-01-11T23:00:11.2261548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2261723Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2262103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2262297Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2262660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2262831Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2263207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2263399Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2263744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2263914Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2264282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2264469Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2264891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2265061Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2265428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2265613Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2265872Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp88fonn3h 2023-01-11T23:00:11.2266125Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp88fonn3h/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2266381Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1d15c98v 2023-01-11T23:00:11.2266645Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1d15c98v/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2266901Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpguujzyj7 2023-01-11T23:00:11.2267168Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpguujzyj7/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2267420Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzd7_gjyp 2023-01-11T23:00:11.2267682Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzd7_gjyp/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2267953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2268170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2268398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2268677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2268827Z fi_getinfo: -61 2023-01-11T23:00:11.2268967Z fi_getinfo: -61 2023-01-11T23:00:11.2269100Z fi_getinfo: -61 2023-01-11T23:00:11.2269235Z fi_getinfo: -61 2023-01-11T23:00:11.2269318Z ok (6.796s) 2023-01-11T23:00:11.2269354Z 2023-01-11T23:00:11.2269599Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2269713Z Ran 1 test in 6.796s 2023-01-11T23:00:11.2269732Z 2023-01-11T23:00:11.2269822Z OK 2023-01-11T23:00:11.2269841Z 2023-01-11T23:00:11.2269964Z Generating XML reports... 2023-01-11T23:00:11.2270514Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225232.xml 2023-01-11T23:00:11.2270887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2271062Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2271441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2271618Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2271873Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfl2q03ba 2023-01-11T23:00:11.2272143Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfl2q03ba/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2272162Z 2023-01-11T23:00:11.2272270Z Running tests... 2023-01-11T23:00:11.2272532Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2272891Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2273194Z test_device_maps_multi_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2273413Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99021 2023-01-11T23:00:11.2273629Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99022 2023-01-11T23:00:11.2273883Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 99023 2023-01-11T23:00:11.2274094Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 99024 2023-01-11T23:00:11.2274468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2274641Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2275021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2275212Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2275572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2275743Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2276098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2276291Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2276650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2276820Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2277235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2277428Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2277798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2277969Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2278335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2278508Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2278766Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2cg8mrgv 2023-01-11T23:00:11.2279034Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2cg8mrgv/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2279291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5t3szih6 2023-01-11T23:00:11.2279561Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5t3szih6/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2279812Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0mhh9oth 2023-01-11T23:00:11.2280078Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0mhh9oth/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2280327Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpon0yglfo 2023-01-11T23:00:11.2280578Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpon0yglfo/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2280807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2281034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2281260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2281487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2281634Z fi_getinfo: -61 2023-01-11T23:00:11.2281773Z fi_getinfo: -61 2023-01-11T23:00:11.2281908Z fi_getinfo: -61 2023-01-11T23:00:11.2282027Z fi_getinfo: -61 2023-01-11T23:00:11.2282126Z ok (10.330s) 2023-01-11T23:00:11.2282145Z 2023-01-11T23:00:11.2282406Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2282521Z Ran 1 test in 10.330s 2023-01-11T23:00:11.2282589Z 2023-01-11T23:00:11.2282682Z OK 2023-01-11T23:00:11.2282701Z 2023-01-11T23:00:11.2282824Z Generating XML reports... 2023-01-11T23:00:11.2283373Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225242.xml 2023-01-11T23:00:11.2283744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2283905Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2284291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2284480Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2284733Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ezaflsc 2023-01-11T23:00:11.2285000Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ezaflsc/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2285022Z 2023-01-11T23:00:11.2285129Z Running tests... 2023-01-11T23:00:11.2285390Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2285751Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2286063Z test_device_maps_multi_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2286313Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99380 2023-01-11T23:00:11.2286538Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99381 2023-01-11T23:00:11.2286751Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 99382 2023-01-11T23:00:11.2286963Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 99383 2023-01-11T23:00:11.2287337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2287514Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2287894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2288085Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2288433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2288604Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2288977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2289166Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2289527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2289701Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2290071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2290256Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2290927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2291090Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2291462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2291646Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2291904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp64d9jnge 2023-01-11T23:00:11.2292174Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp64d9jnge/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2292508Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1i1agm19 2023-01-11T23:00:11.2292775Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1i1agm19/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2293030Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzfla8ghw 2023-01-11T23:00:11.2293282Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzfla8ghw/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2293532Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprrxp86g9 2023-01-11T23:00:11.2293797Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprrxp86g9/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2294029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2294255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2294484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2294711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2294861Z fi_getinfo: -61 2023-01-11T23:00:11.2294982Z fi_getinfo: -61 2023-01-11T23:00:11.2295116Z fi_getinfo: -61 2023-01-11T23:00:11.2295252Z fi_getinfo: -61 2023-01-11T23:00:11.2295351Z ok (10.270s) 2023-01-11T23:00:11.2295479Z 2023-01-11T23:00:11.2295748Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2295861Z Ran 1 test in 10.270s 2023-01-11T23:00:11.2295880Z 2023-01-11T23:00:11.2295971Z OK 2023-01-11T23:00:11.2295991Z 2023-01-11T23:00:11.2296113Z Generating XML reports... 2023-01-11T23:00:11.2296646Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225255.xml 2023-01-11T23:00:11.2297017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2297192Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2297571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2297760Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2298019Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8f6vjyyf 2023-01-11T23:00:11.2298290Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8f6vjyyf/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2298309Z 2023-01-11T23:00:11.2298415Z Running tests... 2023-01-11T23:00:11.2298664Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2299022Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2299329Z test_device_maps_one_to_many (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2299548Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99731 2023-01-11T23:00:11.2299767Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99732 2023-01-11T23:00:11.2299981Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 99733 2023-01-11T23:00:11.2300194Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 99734 2023-01-11T23:00:11.2300565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2300738Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2301102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2301348Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2301715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2301885Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2302256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2302447Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2302808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2302978Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2303330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2303518Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2303889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2304060Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2304428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2304657Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2304923Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl78ehox4 2023-01-11T23:00:11.2305193Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl78ehox4/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2305448Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptuvza8xq 2023-01-11T23:00:11.2305701Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptuvza8xq/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2305959Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwtrdtbfl 2023-01-11T23:00:11.2306227Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwtrdtbfl/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2306477Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps6ssdrdo 2023-01-11T23:00:11.2306741Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps6ssdrdo/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2306972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2307201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2307427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2307653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2307787Z fi_getinfo: -61 2023-01-11T23:00:11.2307887Z ok (4.507s) 2023-01-11T23:00:11.2307907Z 2023-01-11T23:00:11.2308170Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2308282Z Ran 1 test in 4.507s 2023-01-11T23:00:11.2308302Z 2023-01-11T23:00:11.2308394Z OK 2023-01-11T23:00:11.2308414Z 2023-01-11T23:00:11.2308537Z Generating XML reports... 2023-01-11T23:00:11.2309095Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225308.xml 2023-01-11T23:00:11.2309468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2309628Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2310007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2310195Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2310504Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe25umewk 2023-01-11T23:00:11.2310772Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe25umewk/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2310791Z 2023-01-11T23:00:11.2310898Z Running tests... 2023-01-11T23:00:11.2311160Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2311522Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2311805Z test_device_maps_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2312022Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99903 2023-01-11T23:00:11.2312237Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99904 2023-01-11T23:00:11.2312453Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 99905 2023-01-11T23:00:11.2312665Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 99906 2023-01-11T23:00:11.2313037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2313211Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2313640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2313835Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2314184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2314354Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2314728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2314921Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2315279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2315450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2315821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2316010Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2316362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2316535Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2316904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2317090Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2317350Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdwlxmvdb 2023-01-11T23:00:11.2317622Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdwlxmvdb/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2317876Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm7ipbv6f 2023-01-11T23:00:11.2318146Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm7ipbv6f/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2318396Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprwegshuk 2023-01-11T23:00:11.2318646Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprwegshuk/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2318897Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgpcmh6nt 2023-01-11T23:00:11.2319161Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgpcmh6nt/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2319457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2319685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2319909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2320131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2320281Z fi_getinfo: -61 2023-01-11T23:00:11.2320404Z fi_getinfo: -61 2023-01-11T23:00:11.2320540Z fi_getinfo: -61 2023-01-11T23:00:11.2320675Z fi_getinfo: -61 2023-01-11T23:00:11.2320774Z ok (10.392s) 2023-01-11T23:00:11.2320793Z 2023-01-11T23:00:11.2321056Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2321168Z Ran 1 test in 10.392s 2023-01-11T23:00:11.2321187Z 2023-01-11T23:00:11.2321279Z OK 2023-01-11T23:00:11.2321299Z 2023-01-11T23:00:11.2321427Z Generating XML reports... 2023-01-11T23:00:11.2321959Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225315.xml 2023-01-11T23:00:11.2322330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2322504Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2322942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2323136Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2323394Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuldxjkpm 2023-01-11T23:00:11.2323664Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuldxjkpm/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2323684Z 2023-01-11T23:00:11.2323794Z Running tests... 2023-01-11T23:00:11.2324039Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2324401Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2324710Z test_device_maps_return_to_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2324929Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100262 2023-01-11T23:00:11.2325151Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100263 2023-01-11T23:00:11.2325368Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 100264 2023-01-11T23:00:11.2325580Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 100265 2023-01-11T23:00:11.2325953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2326131Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2326495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2326683Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2327048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2327224Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2327603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2327789Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2328154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2328324Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2328755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2328945Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2329306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2329477Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2329848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2330035Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2330508Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa_8ovd0m 2023-01-11T23:00:11.2330788Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa_8ovd0m/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2331046Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ned1zfq 2023-01-11T23:00:11.2331297Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ned1zfq/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2331547Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptms8544h 2023-01-11T23:00:11.2331809Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptms8544h/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2332131Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplu_zzj8q 2023-01-11T23:00:11.2332407Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplu_zzj8q/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2332638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2332865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2333094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2333319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2333451Z skip: Need at least 4 CUDA devices (4.474s) 2023-01-11T23:00:11.2333471Z 2023-01-11T23:00:11.2333747Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2333860Z Ran 1 test in 4.474s 2023-01-11T23:00:11.2333879Z 2023-01-11T23:00:11.2333985Z OK (skipped=1) 2023-01-11T23:00:11.2334006Z 2023-01-11T23:00:11.2334129Z Generating XML reports... 2023-01-11T23:00:11.2334683Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225328.xml 2023-01-11T23:00:11.2335053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2335228Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2335597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2335786Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2336041Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa9m_8h8f 2023-01-11T23:00:11.2336306Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa9m_8h8f/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2336329Z 2023-01-11T23:00:11.2336435Z Running tests... 2023-01-11T23:00:11.2336698Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2337058Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2337369Z test_device_maps_return_to_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2337594Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100433 2023-01-11T23:00:11.2337870Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100434 2023-01-11T23:00:11.2338086Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 100435 2023-01-11T23:00:11.2338298Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 100436 2023-01-11T23:00:11.2338678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2338853Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2339234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2339424Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2339788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2339947Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2340319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2340508Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2340871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2341085Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2341465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2341652Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2342020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2342191Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2342548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2342734Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2342990Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7rnloksw 2023-01-11T23:00:11.2343265Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7rnloksw/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2343520Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppu1rrs46 2023-01-11T23:00:11.2343785Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppu1rrs46/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2344037Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg15jj671 2023-01-11T23:00:11.2344297Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg15jj671/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2344548Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp67itdtzq 2023-01-11T23:00:11.2344796Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp67itdtzq/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2345024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2345251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2345481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2345706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2345854Z skip: Need at least 4 CUDA devices (4.382s) 2023-01-11T23:00:11.2345874Z 2023-01-11T23:00:11.2346149Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2346262Z Ran 1 test in 4.382s 2023-01-11T23:00:11.2346329Z 2023-01-11T23:00:11.2346421Z OK (skipped=1) 2023-01-11T23:00:11.2346440Z 2023-01-11T23:00:11.2346564Z Generating XML reports... 2023-01-11T23:00:11.2347116Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225335.xml 2023-01-11T23:00:11.2347487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2347664Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2348045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2348235Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2348489Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc2lddgec 2023-01-11T23:00:11.2348759Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc2lddgec/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2348781Z 2023-01-11T23:00:11.2348872Z Running tests... 2023-01-11T23:00:11.2349137Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2349496Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2349809Z test_device_maps_wrong_worker_name (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2350076Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100604 2023-01-11T23:00:11.2350298Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100605 2023-01-11T23:00:11.2350515Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 100606 2023-01-11T23:00:11.2350726Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 100607 2023-01-11T23:00:11.2351082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2351260Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2351638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2351828Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2352192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2352363Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2352735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2352924Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2353283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2353442Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2353814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2353998Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2354366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2354539Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2354952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2355139Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2355400Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpczmpxwwy 2023-01-11T23:00:11.2355658Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpczmpxwwy/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2355966Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm28wgh1l 2023-01-11T23:00:11.2356233Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm28wgh1l/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2356483Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp767kj8fm 2023-01-11T23:00:11.2356751Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp767kj8fm/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2357003Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq5x4ufh8 2023-01-11T23:00:11.2357267Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq5x4ufh8/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2357496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2357726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2357940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2358164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2358314Z fi_getinfo: -61 2023-01-11T23:00:11.2358452Z fi_getinfo: -61 2023-01-11T23:00:11.2358589Z fi_getinfo: -61 2023-01-11T23:00:11.2358727Z fi_getinfo: -61 2023-01-11T23:00:11.2358825Z ok (4.697s) 2023-01-11T23:00:11.2358890Z 2023-01-11T23:00:11.2359143Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2359255Z Ran 1 test in 4.697s 2023-01-11T23:00:11.2359274Z 2023-01-11T23:00:11.2359366Z OK 2023-01-11T23:00:11.2359386Z 2023-01-11T23:00:11.2359511Z Generating XML reports... 2023-01-11T23:00:11.2360060Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225342.xml 2023-01-11T23:00:11.2360432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2360604Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2360983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2361172Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2361412Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxfj7oyec 2023-01-11T23:00:11.2361682Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxfj7oyec/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2361701Z 2023-01-11T23:00:11.2361808Z Running tests... 2023-01-11T23:00:11.2362070Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2362431Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2362728Z test_device_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2362949Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100791 2023-01-11T23:00:11.2363165Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100792 2023-01-11T23:00:11.2363364Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 100793 2023-01-11T23:00:11.2363580Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 100794 2023-01-11T23:00:11.2363953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2364126Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2364504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2364750Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2365115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2365286Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2365662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2365836Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2366194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2366364Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2366733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2366919Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2367290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2367460Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2367829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2367998Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2368300Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq4zg5hn0 2023-01-11T23:00:11.2368581Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq4zg5hn0/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2368835Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp99fdkmco 2023-01-11T23:00:11.2369104Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp99fdkmco/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2369359Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwi6fn8jb 2023-01-11T23:00:11.2369625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwi6fn8jb/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2369875Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp65pluacd 2023-01-11T23:00:11.2370139Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp65pluacd/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2370566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2370804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2371031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2371255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2371411Z fi_getinfo: -61 2023-01-11T23:00:11.2371551Z fi_getinfo: -61 2023-01-11T23:00:11.2371688Z fi_getinfo: -61 2023-01-11T23:00:11.2371808Z fi_getinfo: -61 2023-01-11T23:00:11.2371942Z On WorkerInfo(id=1, name=worker1): 2023-01-11T23:00:11.2372292Z RuntimeError('Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!') 2023-01-11T23:00:11.2372426Z Traceback (most recent call last): 2023-01-11T23:00:11.2372784Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function 2023-01-11T23:00:11.2372974Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2023-01-11T23:00:11.2373380Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 5954, in _gpu_add_wrong_gpus 2023-01-11T23:00:11.2373504Z return x.cpu() + y.cuda() 2023-01-11T23:00:11.2373733Z RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! 2023-01-11T23:00:11.2373849Z 2023-01-11T23:00:11.2373970Z On WorkerInfo(id=0, name=worker0): 2023-01-11T23:00:11.2374319Z RuntimeError('Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!') 2023-01-11T23:00:11.2374453Z Traceback (most recent call last): 2023-01-11T23:00:11.2374800Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function 2023-01-11T23:00:11.2374989Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2023-01-11T23:00:11.2375390Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 5954, in _gpu_add_wrong_gpus 2023-01-11T23:00:11.2375510Z return x.cpu() + y.cuda() 2023-01-11T23:00:11.2375755Z RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! 2023-01-11T23:00:11.2375775Z 2023-01-11T23:00:11.2375889Z On WorkerInfo(id=3, name=worker3): 2023-01-11T23:00:11.2376236Z RuntimeError('Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!') 2023-01-11T23:00:11.2376372Z Traceback (most recent call last): 2023-01-11T23:00:11.2376719Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function 2023-01-11T23:00:11.2376906Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2023-01-11T23:00:11.2377359Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 5954, in _gpu_add_wrong_gpus 2023-01-11T23:00:11.2377486Z return x.cpu() + y.cuda() 2023-01-11T23:00:11.2377731Z RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! 2023-01-11T23:00:11.2377751Z 2023-01-11T23:00:11.2377880Z On WorkerInfo(id=2, name=worker2): 2023-01-11T23:00:11.2378211Z RuntimeError('Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!') 2023-01-11T23:00:11.2378344Z Traceback (most recent call last): 2023-01-11T23:00:11.2378693Z File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function 2023-01-11T23:00:11.2378879Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2023-01-11T23:00:11.2379279Z File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 5954, in _gpu_add_wrong_gpus 2023-01-11T23:00:11.2379399Z return x.cpu() + y.cuda() 2023-01-11T23:00:11.2379647Z RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! 2023-01-11T23:00:11.2379666Z 2023-01-11T23:00:11.2379767Z ok (7.900s) 2023-01-11T23:00:11.2379786Z 2023-01-11T23:00:11.2380037Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2380148Z Ran 1 test in 7.900s 2023-01-11T23:00:11.2380166Z 2023-01-11T23:00:11.2380260Z OK 2023-01-11T23:00:11.2380279Z 2023-01-11T23:00:11.2380402Z Generating XML reports... 2023-01-11T23:00:11.2380955Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225350.xml 2023-01-11T23:00:11.2381323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2381498Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2381882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2382075Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2382314Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7gsdc2xv 2023-01-11T23:00:11.2382587Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7gsdc2xv/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2382607Z 2023-01-11T23:00:11.2382716Z Running tests... 2023-01-11T23:00:11.2383032Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2383392Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2383698Z test_devices_option_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2383920Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101138 2023-01-11T23:00:11.2384143Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101139 2023-01-11T23:00:11.2384342Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 101140 2023-01-11T23:00:11.2384557Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 101141 2023-01-11T23:00:11.2384928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2385104Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2385488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2385677Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2386046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2386219Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2386651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2386829Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2387195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2387371Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2387744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2387933Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2388294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2388465Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2388847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2389014Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2389271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4mdv8pej 2023-01-11T23:00:11.2389543Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4mdv8pej/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2389798Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpilo_xtrs 2023-01-11T23:00:11.2390069Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpilo_xtrs/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2390321Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprgf2jcv7 2023-01-11T23:00:11.2390589Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprgf2jcv7/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2390844Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwfgiafa0 2023-01-11T23:00:11.2391112Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwfgiafa0/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2391325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2391546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2391775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2392056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2392203Z fi_getinfo: -61 2023-01-11T23:00:11.2392339Z fi_getinfo: -61 2023-01-11T23:00:11.2392473Z fi_getinfo: -61 2023-01-11T23:00:11.2392592Z fi_getinfo: -61 2023-01-11T23:00:11.2392691Z ok (4.625s) 2023-01-11T23:00:11.2392710Z 2023-01-11T23:00:11.2392971Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2393086Z Ran 1 test in 4.625s 2023-01-11T23:00:11.2393105Z 2023-01-11T23:00:11.2393197Z OK 2023-01-11T23:00:11.2393215Z 2023-01-11T23:00:11.2393338Z Generating XML reports... 2023-01-11T23:00:11.2393887Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225400.xml 2023-01-11T23:00:11.2394258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2394436Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2394800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2394989Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2395240Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7_sxgljt 2023-01-11T23:00:11.2395555Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7_sxgljt/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2395575Z 2023-01-11T23:00:11.2395686Z Running tests... 2023-01-11T23:00:11.2395948Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2396305Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2396623Z test_devices_option_mismatch_reverse (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2396830Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101325 2023-01-11T23:00:11.2397047Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101326 2023-01-11T23:00:11.2397261Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 101327 2023-01-11T23:00:11.2397473Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 101328 2023-01-11T23:00:11.2397848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2398022Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2398403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2398592Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2398955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2399113Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2399484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2399670Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2400042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2400212Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2400582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2400772Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2401133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2401340Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2401711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2401894Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2402150Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg_fkfmuk 2023-01-11T23:00:11.2402424Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg_fkfmuk/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2402674Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_tybu916 2023-01-11T23:00:11.2402940Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_tybu916/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2403192Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1xoh0ue5 2023-01-11T23:00:11.2403462Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1xoh0ue5/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2403698Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1yp5d6fl 2023-01-11T23:00:11.2403964Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1yp5d6fl/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2404193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2404461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2404694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2404920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2405064Z fi_getinfo: -61 2023-01-11T23:00:11.2405200Z fi_getinfo: -61 2023-01-11T23:00:11.2405321Z fi_getinfo: -61 2023-01-11T23:00:11.2405456Z fi_getinfo: -61 2023-01-11T23:00:11.2405558Z ok (4.616s) 2023-01-11T23:00:11.2405578Z 2023-01-11T23:00:11.2405841Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2405952Z Ran 1 test in 4.617s 2023-01-11T23:00:11.2405971Z 2023-01-11T23:00:11.2406063Z OK 2023-01-11T23:00:11.2406083Z 2023-01-11T23:00:11.2406206Z Generating XML reports... 2023-01-11T23:00:11.2406756Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225408.xml 2023-01-11T23:00:11.2407112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2407287Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2407667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2407859Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2408116Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbxm1mrd8 2023-01-11T23:00:11.2408387Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbxm1mrd8/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2408406Z 2023-01-11T23:00:11.2408514Z Running tests... 2023-01-11T23:00:11.2408776Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2409124Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2409450Z test_owner_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2409669Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101512 2023-01-11T23:00:11.2409887Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101513 2023-01-11T23:00:11.2410100Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 101514 2023-01-11T23:00:11.2410578Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 101515 2023-01-11T23:00:11.2410966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2411143Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2411527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2411702Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2412063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2412234Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2412607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2412799Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2413160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2413330Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2413701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2413939Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2414318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2414488Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2414856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2415042Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2415301Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprxo4tiij 2023-01-11T23:00:11.2415573Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprxo4tiij/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2415828Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyt7ffv6d 2023-01-11T23:00:11.2416099Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyt7ffv6d/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2416335Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3mkkgoz5 2023-01-11T23:00:11.2416582Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9rhxn1dh 2023-01-11T23:00:11.2416846Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3mkkgoz5/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2417110Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9rhxn1dh/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2417344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2417571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2417798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2418022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2418154Z fi_getinfo: -61 2023-01-11T23:00:11.2418255Z ok (10.826s) 2023-01-11T23:00:11.2418274Z 2023-01-11T23:00:11.2418538Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2418651Z Ran 1 test in 10.826s 2023-01-11T23:00:11.2418671Z 2023-01-11T23:00:11.2418763Z OK 2023-01-11T23:00:11.2418782Z 2023-01-11T23:00:11.2418912Z Generating XML reports... 2023-01-11T23:00:11.2419461Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225415.xml 2023-01-11T23:00:11.2419905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2420079Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2420445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2420637Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2420893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgj0gzvtz 2023-01-11T23:00:11.2421164Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgj0gzvtz/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2421185Z 2023-01-11T23:00:11.2421293Z Running tests... 2023-01-11T23:00:11.2421554Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2421915Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2422240Z test_owner_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2422444Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101727 2023-01-11T23:00:11.2422664Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101728 2023-01-11T23:00:11.2422923Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 101729 2023-01-11T23:00:11.2423141Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 101730 2023-01-11T23:00:11.2423514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2423690Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2424069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2424262Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2424627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2424783Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2425162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2425350Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2425711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2425880Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2426249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2426440Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2426808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2426961Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2427333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2427519Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2427779Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp81yr2h47 2023-01-11T23:00:11.2428046Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp81yr2h47/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2428301Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk954k014 2023-01-11T23:00:11.2428617Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk954k014/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2428868Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp87mexc2_ 2023-01-11T23:00:11.2429131Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp87mexc2_/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2429362Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkj92xw8n 2023-01-11T23:00:11.2429628Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkj92xw8n/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2429859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2430085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2430311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2430534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2430688Z fi_getinfo: -61 2023-01-11T23:00:11.2430787Z ok (12.228s) 2023-01-11T23:00:11.2430806Z 2023-01-11T23:00:11.2431052Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2431165Z Ran 1 test in 12.228s 2023-01-11T23:00:11.2431184Z 2023-01-11T23:00:11.2431275Z OK 2023-01-11T23:00:11.2431294Z 2023-01-11T23:00:11.2431418Z Generating XML reports... 2023-01-11T23:00:11.2432007Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225429.xml 2023-01-11T23:00:11.2432385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2432560Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2432939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2433132Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2433370Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt6s4mm7o 2023-01-11T23:00:11.2433638Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt6s4mm7o/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2433657Z 2023-01-11T23:00:11.2433764Z Running tests... 2023-01-11T23:00:11.2434029Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2434385Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2434709Z test_owner_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2434929Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101943 2023-01-11T23:00:11.2435149Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101944 2023-01-11T23:00:11.2435349Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 101945 2023-01-11T23:00:11.2435560Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 101946 2023-01-11T23:00:11.2435934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2436108Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2436490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2436679Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2437042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2437214Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2437699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2437871Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2438232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2438406Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2438778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2438964Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2439328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2439497Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2439864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2440036Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2440294Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0j1vvpl7 2023-01-11T23:00:11.2440565Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0j1vvpl7/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2440864Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprg6786m5 2023-01-11T23:00:11.2441135Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprg6786m5/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2441389Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt5pzynmr 2023-01-11T23:00:11.2441654Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt5pzynmr/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2441904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf1mdvg_i 2023-01-11T23:00:11.2442175Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf1mdvg_i/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2442389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2442616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2442843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2443070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2443217Z fi_getinfo: -61 2023-01-11T23:00:11.2443319Z ok (12.484s) 2023-01-11T23:00:11.2443339Z 2023-01-11T23:00:11.2443600Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2443713Z Ran 1 test in 12.484s 2023-01-11T23:00:11.2443732Z 2023-01-11T23:00:11.2443806Z OK 2023-01-11T23:00:11.2443824Z 2023-01-11T23:00:11.2443951Z Generating XML reports... 2023-01-11T23:00:11.2444497Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225444.xml 2023-01-11T23:00:11.2444867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2445041Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2445426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2445619Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2445874Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphxiwd6sz 2023-01-11T23:00:11.2446144Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphxiwd6sz/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2446164Z 2023-01-11T23:00:11.2446256Z Running tests... 2023-01-11T23:00:11.2446572Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2446934Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2447259Z test_owner_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2447481Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102159 2023-01-11T23:00:11.2447701Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102160 2023-01-11T23:00:11.2447915Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 102161 2023-01-11T23:00:11.2448127Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 102162 2023-01-11T23:00:11.2448480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2448658Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2449034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2449226Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2449591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2449816Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2450195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2450594Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2450968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2451122Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2451496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2451683Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2452047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2452216Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2452588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2452773Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2453030Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpufet36oa 2023-01-11T23:00:11.2453286Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpufet36oa/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2453544Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo2lnqfve 2023-01-11T23:00:11.2453812Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo2lnqfve/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2454064Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprlg56t3p 2023-01-11T23:00:11.2454330Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprlg56t3p/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2454581Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw_ul3l_b 2023-01-11T23:00:11.2454879Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw_ul3l_b/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2455112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2455339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2455549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2455855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2456004Z fi_getinfo: -61 2023-01-11T23:00:11.2456105Z ok (10.614s) 2023-01-11T23:00:11.2456124Z 2023-01-11T23:00:11.2456387Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2456500Z Ran 1 test in 10.614s 2023-01-11T23:00:11.2456519Z 2023-01-11T23:00:11.2456610Z OK 2023-01-11T23:00:11.2456634Z 2023-01-11T23:00:11.2456758Z Generating XML reports... 2023-01-11T23:00:11.2457292Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225459.xml 2023-01-11T23:00:11.2457662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2457836Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2458220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2458410Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2458665Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8mf_33qj 2023-01-11T23:00:11.2458931Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8mf_33qj/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2459010Z 2023-01-11T23:00:11.2459120Z Running tests... 2023-01-11T23:00:11.2459384Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2459727Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2460041Z test_rref_as_arg_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2460261Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102374 2023-01-11T23:00:11.2460482Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102375 2023-01-11T23:00:11.2460699Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 102376 2023-01-11T23:00:11.2460911Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 102377 2023-01-11T23:00:11.2461288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2461464Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2461825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2462014Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2462378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2462552Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2462924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2463112Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2463471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2463644Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2464013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2464183Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2464547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2464717Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2465151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2465337Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2465596Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm6lg1vua 2023-01-11T23:00:11.2465872Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm6lg1vua/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2466129Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfz407dwn 2023-01-11T23:00:11.2466397Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfz407dwn/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2466635Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfmynlkvh 2023-01-11T23:00:11.2466903Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfmynlkvh/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2467156Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe7w1ixu1 2023-01-11T23:00:11.2467419Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe7w1ixu1/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2467647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2467874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2468146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2468376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2468506Z fi_getinfo: -61 2023-01-11T23:00:11.2468641Z fi_getinfo: -61 2023-01-11T23:00:11.2468774Z fi_getinfo: -61 2023-01-11T23:00:11.2468910Z fi_getinfo: -61 2023-01-11T23:00:11.2469008Z ok (17.709s) 2023-01-11T23:00:11.2469028Z 2023-01-11T23:00:11.2469294Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2469408Z Ran 1 test in 17.709s 2023-01-11T23:00:11.2469428Z 2023-01-11T23:00:11.2469504Z OK 2023-01-11T23:00:11.2469539Z 2023-01-11T23:00:11.2469646Z Generating XML reports... 2023-01-11T23:00:11.2470196Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225512.xml 2023-01-11T23:00:11.2470568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2470744Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2471125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2471314Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2471569Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm9u9pq_d 2023-01-11T23:00:11.2471843Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm9u9pq_d/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2471862Z 2023-01-11T23:00:11.2471953Z Running tests... 2023-01-11T23:00:11.2472212Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2472573Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2472888Z test_rref_as_arg_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2473107Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102721 2023-01-11T23:00:11.2473327Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102722 2023-01-11T23:00:11.2473542Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 102723 2023-01-11T23:00:11.2473753Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 102724 2023-01-11T23:00:11.2474168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2474343Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2474721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2474919Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2475282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2475453Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2475823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2476012Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2476381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2476534Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2476907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2477092Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2477506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2477681Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2478055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2478240Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2478497Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp74l99soj 2023-01-11T23:00:11.2478764Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp74l99soj/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2479004Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphgnzpanz 2023-01-11T23:00:11.2479275Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphgnzpanz/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2479530Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzh71ip7w 2023-01-11T23:00:11.2479799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzh71ip7w/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2480051Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyd5f2szd 2023-01-11T23:00:11.2480315Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyd5f2szd/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2480546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2480776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2480984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2481210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2481356Z fi_getinfo: -61 2023-01-11T23:00:11.2481494Z fi_getinfo: -61 2023-01-11T23:00:11.2481633Z fi_getinfo: -61 2023-01-11T23:00:11.2481770Z fi_getinfo: -61 2023-01-11T23:00:11.2481867Z ok (20.197s) 2023-01-11T23:00:11.2481887Z 2023-01-11T23:00:11.2482133Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2482246Z Ran 1 test in 20.197s 2023-01-11T23:00:11.2482265Z 2023-01-11T23:00:11.2482356Z OK 2023-01-11T23:00:11.2482377Z 2023-01-11T23:00:11.2482501Z Generating XML reports... 2023-01-11T23:00:11.2483051Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225533.xml 2023-01-11T23:00:11.2483475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2483649Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2484029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2484219Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2484453Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_cgj31as 2023-01-11T23:00:11.2484721Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_cgj31as/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2484741Z 2023-01-11T23:00:11.2484848Z Running tests... 2023-01-11T23:00:11.2485108Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2485473Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2485786Z test_rref_as_arg_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2486598Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81962 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.566s) 2023-01-11T23:00:11.2486620Z 2023-01-11T23:00:11.2486885Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2486995Z Ran 1 test in 1.566s 2023-01-11T23:00:11.2487014Z 2023-01-11T23:00:11.2487120Z OK (skipped=1) 2023-01-11T23:00:11.2487139Z 2023-01-11T23:00:11.2487246Z Generating XML reports... 2023-01-11T23:00:11.2487801Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225556.xml 2023-01-11T23:00:11.2488171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2488349Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2488733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2488924Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2489180Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv6mfze8z 2023-01-11T23:00:11.2489449Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv6mfze8z/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2489468Z 2023-01-11T23:00:11.2489575Z Running tests... 2023-01-11T23:00:11.2489817Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2490181Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2490706Z test_rref_as_arg_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2490937Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103108 2023-01-11T23:00:11.2491162Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103109 2023-01-11T23:00:11.2491380Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 103110 2023-01-11T23:00:11.2491591Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 103111 2023-01-11T23:00:11.2491969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2492127Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2492589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2492780Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2493143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2493314Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2493687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2493873Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2494237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2494407Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2494759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2494946Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2495311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2495483Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2495903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2496093Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2496352Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpul7qny1r 2023-01-11T23:00:11.2496622Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpul7qny1r/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2496858Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp452y5vnh 2023-01-11T23:00:11.2497132Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp452y5vnh/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2497384Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprhny3dev 2023-01-11T23:00:11.2497653Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprhny3dev/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2497908Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr2srykxw 2023-01-11T23:00:11.2498173Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr2srykxw/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2498401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2498629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2498854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2499067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2499214Z fi_getinfo: -61 2023-01-11T23:00:11.2499351Z fi_getinfo: -61 2023-01-11T23:00:11.2499485Z fi_getinfo: -61 2023-01-11T23:00:11.2499620Z fi_getinfo: -61 2023-01-11T23:00:11.2499718Z ok (20.223s) 2023-01-11T23:00:11.2499737Z 2023-01-11T23:00:11.2499997Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2500094Z Ran 1 test in 20.223s 2023-01-11T23:00:11.2500116Z 2023-01-11T23:00:11.2500207Z OK 2023-01-11T23:00:11.2500226Z 2023-01-11T23:00:11.2500348Z Generating XML reports... 2023-01-11T23:00:11.2500900Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225600.xml 2023-01-11T23:00:11.2501270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2501502Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2501883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2502071Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2502309Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu_9oy_h9 2023-01-11T23:00:11.2502581Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu_9oy_h9/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2502601Z 2023-01-11T23:00:11.2502709Z Running tests... 2023-01-11T23:00:11.2502969Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2503327Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2503639Z test_rref_as_arg_synchronization5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2503864Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103461 2023-01-11T23:00:11.2504081Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103462 2023-01-11T23:00:11.2504297Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 103463 2023-01-11T23:00:11.2504492Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 103464 2023-01-11T23:00:11.2504930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2505113Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2505482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2505654Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2506030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2506223Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2506597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2506787Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2507138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2507312Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2507683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2507867Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2508233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2508406Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2508775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2508961Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2509199Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz8zhxemu 2023-01-11T23:00:11.2509471Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz8zhxemu/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2509726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp36ksxowe 2023-01-11T23:00:11.2509994Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp36ksxowe/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2510246Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4tkpwobm 2023-01-11T23:00:11.2510512Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4tkpwobm/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2510818Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk75v0_t8 2023-01-11T23:00:11.2511078Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk75v0_t8/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2511307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2511522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2511750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2511975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2512122Z fi_getinfo: -61 2023-01-11T23:00:11.2512259Z fi_getinfo: -61 2023-01-11T23:00:11.2512393Z fi_getinfo: -61 2023-01-11T23:00:11.2512528Z fi_getinfo: -61 2023-01-11T23:00:11.2512613Z ok (17.732s) 2023-01-11T23:00:11.2512633Z 2023-01-11T23:00:11.2512895Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2513009Z Ran 1 test in 17.732s 2023-01-11T23:00:11.2513028Z 2023-01-11T23:00:11.2513120Z OK 2023-01-11T23:00:11.2513139Z 2023-01-11T23:00:11.2513262Z Generating XML reports... 2023-01-11T23:00:11.2513866Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225623.xml 2023-01-11T23:00:11.2514245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2514421Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2514801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2514975Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2515233Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpucfqunrb 2023-01-11T23:00:11.2515506Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpucfqunrb/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2515525Z 2023-01-11T23:00:11.2515634Z Running tests... 2023-01-11T23:00:11.2515894Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2516255Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2516570Z test_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2516789Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103808 2023-01-11T23:00:11.2516988Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103809 2023-01-11T23:00:11.2517202Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 103810 2023-01-11T23:00:11.2517415Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 103811 2023-01-11T23:00:11.2517791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2517966Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2518347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2518539Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2518901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2519073Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2519429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2519671Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2520031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2520202Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2520573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2520761Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2521128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2521299Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2521651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2521836Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2522097Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx0wbqams 2023-01-11T23:00:11.2522366Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx0wbqams/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2522619Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpucls1lvo 2023-01-11T23:00:11.2522930Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpucls1lvo/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2523188Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvntq93ph 2023-01-11T23:00:11.2523455Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvntq93ph/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2523704Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp61p8uafo 2023-01-11T23:00:11.2523952Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp61p8uafo/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2524187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2524413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2524645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2524789Z fi_getinfo: -61 2023-01-11T23:00:11.2525016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2525156Z fi_getinfo: -61 2023-01-11T23:00:11.2525291Z fi_getinfo: -61 2023-01-11T23:00:11.2525410Z fi_getinfo: -61 2023-01-11T23:00:11.2525509Z ok (16.520s) 2023-01-11T23:00:11.2525528Z 2023-01-11T23:00:11.2525788Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2525901Z Ran 1 test in 16.520s 2023-01-11T23:00:11.2525921Z 2023-01-11T23:00:11.2526013Z OK 2023-01-11T23:00:11.2526032Z 2023-01-11T23:00:11.2526158Z Generating XML reports... 2023-01-11T23:00:11.2526707Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225644.xml 2023-01-11T23:00:11.2527076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2527233Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2527618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2527808Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2528060Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_6v1i1c0 2023-01-11T23:00:11.2528322Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_6v1i1c0/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2528341Z 2023-01-11T23:00:11.2528448Z Running tests... 2023-01-11T23:00:11.2528763Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2529123Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2529424Z test_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2529643Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104154 2023-01-11T23:00:11.2529866Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104155 2023-01-11T23:00:11.2530082Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 104156 2023-01-11T23:00:11.2530510Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 104157 2023-01-11T23:00:11.2530903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2531078Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2531464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2531656Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2532005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2532245Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2532631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2532820Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2533180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2533349Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2533726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2533911Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2534261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2534433Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2534805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2534990Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2535245Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8c3ifai5 2023-01-11T23:00:11.2535513Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8c3ifai5/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2535768Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp7950b85 2023-01-11T23:00:11.2536036Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp7950b85/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2536288Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbkm7cowk 2023-01-11T23:00:11.2536539Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbkm7cowk/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2536795Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5cp209q8 2023-01-11T23:00:11.2537058Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5cp209q8/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2537288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2537514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2537739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2538039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2538185Z fi_getinfo: -61 2023-01-11T23:00:11.2538306Z fi_getinfo: -61 2023-01-11T23:00:11.2538441Z fi_getinfo: -61 2023-01-11T23:00:11.2538577Z fi_getinfo: -61 2023-01-11T23:00:11.2538677Z ok (16.738s) 2023-01-11T23:00:11.2538697Z 2023-01-11T23:00:11.2538957Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2539073Z Ran 1 test in 16.739s 2023-01-11T23:00:11.2539093Z 2023-01-11T23:00:11.2539184Z OK 2023-01-11T23:00:11.2539203Z 2023-01-11T23:00:11.2539309Z Generating XML reports... 2023-01-11T23:00:11.2539858Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225703.xml 2023-01-11T23:00:11.2540230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2540407Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2540785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2540974Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2541228Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3fbiuodc 2023-01-11T23:00:11.2541545Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3fbiuodc/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2541566Z 2023-01-11T23:00:11.2541676Z Running tests... 2023-01-11T23:00:11.2541921Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2542281Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2542596Z test_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2542819Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104503 2023-01-11T23:00:11.2543037Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104504 2023-01-11T23:00:11.2543253Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 104505 2023-01-11T23:00:11.2543463Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 104506 2023-01-11T23:00:11.2543838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2544012Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2544376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2544567Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2544935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2545105Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2545476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2545664Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2546027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2546196Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2546551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2546740Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2547108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2547329Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2547701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2547885Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2548143Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9m0qplw8 2023-01-11T23:00:11.2548412Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9m0qplw8/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2548668Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq8y253t0 2023-01-11T23:00:11.2548918Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq8y253t0/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2549170Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpehe84yht 2023-01-11T23:00:11.2549439Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpehe84yht/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2549689Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpug5nocnb 2023-01-11T23:00:11.2549949Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpug5nocnb/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2550178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2550450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2550680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2550889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2551037Z fi_getinfo: -61 2023-01-11T23:00:11.2551171Z fi_getinfo: -61 2023-01-11T23:00:11.2551305Z fi_getinfo: -61 2023-01-11T23:00:11.2551444Z fi_getinfo: -61 2023-01-11T23:00:11.2551542Z ok (16.419s) 2023-01-11T23:00:11.2551561Z 2023-01-11T23:00:11.2551824Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2551921Z Ran 1 test in 16.419s 2023-01-11T23:00:11.2551957Z 2023-01-11T23:00:11.2552032Z OK 2023-01-11T23:00:11.2552050Z 2023-01-11T23:00:11.2552173Z Generating XML reports... 2023-01-11T23:00:11.2552727Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225722.xml 2023-01-11T23:00:11.2553097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2553271Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2553649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2553839Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2554095Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6xe32gdd 2023-01-11T23:00:11.2554347Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6xe32gdd/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2554382Z 2023-01-11T23:00:11.2554472Z Running tests... 2023-01-11T23:00:11.2554734Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2555142Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2555459Z test_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2555679Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104852 2023-01-11T23:00:11.2555897Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104853 2023-01-11T23:00:11.2556111Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 104854 2023-01-11T23:00:11.2556376Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 104855 2023-01-11T23:00:11.2556735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2556908Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2557288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2557478Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2557840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2558010Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2558383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2558575Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2558922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2559094Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2559509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2559704Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2560071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2560241Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2560609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2560798Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2561053Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr5fnkeyc 2023-01-11T23:00:11.2561309Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr5fnkeyc/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2561564Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4hipx_um 2023-01-11T23:00:11.2561835Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4hipx_um/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2562087Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdsr3ze_2 2023-01-11T23:00:11.2562355Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdsr3ze_2/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2562606Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpznskthc4 2023-01-11T23:00:11.2562871Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpznskthc4/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2563104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2563314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2563541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2563769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2563916Z fi_getinfo: -61 2023-01-11T23:00:11.2564055Z fi_getinfo: -61 2023-01-11T23:00:11.2564190Z fi_getinfo: -61 2023-01-11T23:00:11.2564324Z fi_getinfo: -61 2023-01-11T23:00:11.2564423Z ok (16.425s) 2023-01-11T23:00:11.2564443Z 2023-01-11T23:00:11.2564689Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2564802Z Ran 1 test in 16.425s 2023-01-11T23:00:11.2564821Z 2023-01-11T23:00:11.2564911Z OK 2023-01-11T23:00:11.2564979Z 2023-01-11T23:00:11.2565106Z Generating XML reports... 2023-01-11T23:00:11.2565657Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225742.xml 2023-01-11T23:00:11.2566024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2566199Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2566581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2566755Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2567010Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeww47b_l 2023-01-11T23:00:11.2567279Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeww47b_l/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2567301Z 2023-01-11T23:00:11.2567409Z Running tests... 2023-01-11T23:00:11.2567668Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2568027Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2568341Z test_rref_to_here_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2568664Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105198 2023-01-11T23:00:11.2568893Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105199 2023-01-11T23:00:11.2569092Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 105200 2023-01-11T23:00:11.2569303Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 105201 2023-01-11T23:00:11.2569684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2569867Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2570248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2570639Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2571013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2571191Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2571550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2571740Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2572101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2572271Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2572642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2572829Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2573193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2573368Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2573735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2573905Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2574162Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp45ghi53j 2023-01-11T23:00:11.2574431Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp45ghi53j/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2574770Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpobc_n1x7 2023-01-11T23:00:11.2575037Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpobc_n1x7/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2575292Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxkdmpbf8 2023-01-11T23:00:11.2575561Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxkdmpbf8/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2575815Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpavufi0cl 2023-01-11T23:00:11.2576064Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpavufi0cl/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2576294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2576521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2576748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2576973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2577120Z fi_getinfo: -61 2023-01-11T23:00:11.2577256Z fi_getinfo: -61 2023-01-11T23:00:11.2577392Z fi_getinfo: -61 2023-01-11T23:00:11.2577510Z fi_getinfo: -61 2023-01-11T23:00:11.2577609Z ok (17.758s) 2023-01-11T23:00:11.2577628Z 2023-01-11T23:00:11.2578014Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2578135Z Ran 1 test in 17.759s 2023-01-11T23:00:11.2578155Z 2023-01-11T23:00:11.2578248Z OK 2023-01-11T23:00:11.2578267Z 2023-01-11T23:00:11.2578390Z Generating XML reports... 2023-01-11T23:00:11.2578943Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225801.xml 2023-01-11T23:00:11.2579314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2579477Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2579858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2580049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2580307Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpden4zjog 2023-01-11T23:00:11.2580577Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpden4zjog/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2580596Z 2023-01-11T23:00:11.2580703Z Running tests... 2023-01-11T23:00:11.2580964Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2581323Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2581638Z test_rref_to_here_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2581847Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105545 2023-01-11T23:00:11.2582063Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105546 2023-01-11T23:00:11.2582278Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 105547 2023-01-11T23:00:11.2582492Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 105548 2023-01-11T23:00:11.2582866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2583040Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2583420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2583610Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2584014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2584186Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2584556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2584746Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2585108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2585280Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2585652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2585838Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2586209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2586363Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2586735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2586920Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2587222Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqclonr7u 2023-01-11T23:00:11.2587500Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqclonr7u/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2587755Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi3av82z2 2023-01-11T23:00:11.2588019Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi3av82z2/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2588270Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd35k2wdi 2023-01-11T23:00:11.2588539Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd35k2wdi/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2588772Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp67kh5fbo 2023-01-11T23:00:11.2589035Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp67kh5fbo/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2589266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2589494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2589720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2589945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2590092Z fi_getinfo: -61 2023-01-11T23:00:11.2590230Z fi_getinfo: -61 2023-01-11T23:00:11.2590356Z fi_getinfo: -61 2023-01-11T23:00:11.2590490Z fi_getinfo: -61 2023-01-11T23:00:11.2590588Z ok (20.250s) 2023-01-11T23:00:11.2590607Z 2023-01-11T23:00:11.2590868Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2590979Z Ran 1 test in 20.250s 2023-01-11T23:00:11.2590998Z 2023-01-11T23:00:11.2591090Z OK 2023-01-11T23:00:11.2591109Z 2023-01-11T23:00:11.2591231Z Generating XML reports... 2023-01-11T23:00:11.2591769Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225821.xml 2023-01-11T23:00:11.2592140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2592314Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2592693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2592935Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2593190Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbgi34yku 2023-01-11T23:00:11.2593459Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbgi34yku/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2593479Z 2023-01-11T23:00:11.2593586Z Running tests... 2023-01-11T23:00:11.2593850Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2594196Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2594512Z test_rref_to_here_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2594731Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105898 2023-01-11T23:00:11.2594951Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105899 2023-01-11T23:00:11.2595172Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 105900 2023-01-11T23:00:11.2595383Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 105901 2023-01-11T23:00:11.2595756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2595929Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2596340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2596537Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2596906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2597076Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2597449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2597639Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2597999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2598168Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2598540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2598708Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2599075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2599244Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2599613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2599800Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2600057Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptjhttfkj 2023-01-11T23:00:11.2600332Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptjhttfkj/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2600588Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc_g21bz8 2023-01-11T23:00:11.2600857Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc_g21bz8/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2601094Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvf41myqj 2023-01-11T23:00:11.2601358Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvf41myqj/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2601610Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc2d2v9x5 2023-01-11T23:00:11.2601928Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc2d2v9x5/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2602156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2602387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2602613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2602842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2602973Z fi_getinfo: -61 2023-01-11T23:00:11.2603112Z fi_getinfo: -61 2023-01-11T23:00:11.2603247Z fi_getinfo: -61 2023-01-11T23:00:11.2603385Z fi_getinfo: -61 2023-01-11T23:00:11.2603485Z ok (17.690s) 2023-01-11T23:00:11.2603505Z 2023-01-11T23:00:11.2603768Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2603880Z Ran 1 test in 17.690s 2023-01-11T23:00:11.2603902Z 2023-01-11T23:00:11.2603978Z OK 2023-01-11T23:00:11.2604012Z 2023-01-11T23:00:11.2604119Z Generating XML reports... 2023-01-11T23:00:11.2604670Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225844.xml 2023-01-11T23:00:11.2605041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2605265Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2605655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2605846Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2606103Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpydcdhfnw 2023-01-11T23:00:11.2606376Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpydcdhfnw/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2606398Z 2023-01-11T23:00:11.2606491Z Running tests... 2023-01-11T23:00:11.2606754Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2607112Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2607430Z test_rref_to_here_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2607652Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106245 2023-01-11T23:00:11.2607870Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106246 2023-01-11T23:00:11.2608086Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 106247 2023-01-11T23:00:11.2608297Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 106248 2023-01-11T23:00:11.2608652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2608829Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2609211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2609403Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2609771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2609942Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2610515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2610717Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2611082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2611318Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2611692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2611879Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2612247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2612422Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2612792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2612978Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2613236Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphfardwm0 2023-01-11T23:00:11.2613512Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphfardwm0/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2613754Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0xfbrmb1 2023-01-11T23:00:11.2614023Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0xfbrmb1/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2614278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqfpr8irr 2023-01-11T23:00:11.2614602Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqfpr8irr/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2614858Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp99wqm_zz 2023-01-11T23:00:11.2615121Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp99wqm_zz/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2615352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2615579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2615794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2616020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2616175Z fi_getinfo: -61 2023-01-11T23:00:11.2616312Z fi_getinfo: -61 2023-01-11T23:00:11.2616447Z fi_getinfo: -61 2023-01-11T23:00:11.2616580Z fi_getinfo: -61 2023-01-11T23:00:11.2616679Z ok (20.196s) 2023-01-11T23:00:11.2616702Z 2023-01-11T23:00:11.2616949Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2617062Z Ran 1 test in 20.197s 2023-01-11T23:00:11.2617081Z 2023-01-11T23:00:11.2617172Z OK 2023-01-11T23:00:11.2617191Z 2023-01-11T23:00:11.2617314Z Generating XML reports... 2023-01-11T23:00:11.2617865Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225905.xml 2023-01-11T23:00:11.2618237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2618411Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2618790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2618983Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2619223Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3hh_csww 2023-01-11T23:00:11.2619491Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3hh_csww/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2619510Z 2023-01-11T23:00:11.2619618Z Running tests... 2023-01-11T23:00:11.2619877Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2620238Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2620616Z test_rref_with_unpickleable_attributes (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2620837Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106598 2023-01-11T23:00:11.2621054Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106599 2023-01-11T23:00:11.2621269Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 106600 2023-01-11T23:00:11.2621466Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 106601 2023-01-11T23:00:11.2621842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2622016Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2622396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2622588Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2622951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2623126Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2623499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2623718Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2624087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2624259Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2624627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2624813Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2625186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2625358Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2625727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2625917Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2626162Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnebw7c08 2023-01-11T23:00:11.2626431Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnebw7c08/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2626685Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv5nplbza 2023-01-11T23:00:11.2626955Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv5nplbza/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2627208Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7v92aeoo 2023-01-11T23:00:11.2627471Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7v92aeoo/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2627718Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps_kzck_x 2023-01-11T23:00:11.2627982Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps_kzck_x/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2628198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2628426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2628652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2628878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2629025Z fi_getinfo: -61 2023-01-11T23:00:11.2629216Z fi_getinfo: -61 2023-01-11T23:00:11.2629353Z fi_getinfo: -61 2023-01-11T23:00:11.2629489Z fi_getinfo: -61 2023-01-11T23:00:11.2629573Z ok (7.916s) 2023-01-11T23:00:11.2629592Z 2023-01-11T23:00:11.2629854Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2629966Z Ran 1 test in 7.917s 2023-01-11T23:00:11.2629985Z 2023-01-11T23:00:11.2630078Z OK 2023-01-11T23:00:11.2630096Z 2023-01-11T23:00:11.2630218Z Generating XML reports... 2023-01-11T23:00:11.2630770Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225928.xml 2023-01-11T23:00:11.2631140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2631314Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2631677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2631870Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2632126Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgmfc1z_f 2023-01-11T23:00:11.2632398Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgmfc1z_f/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2632417Z 2023-01-11T23:00:11.2632524Z Running tests... 2023-01-11T23:00:11.2632827Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2633192Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2633501Z test_tensor_view_as_return_value (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2633719Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106949 2023-01-11T23:00:11.2633924Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106950 2023-01-11T23:00:11.2634143Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 106951 2023-01-11T23:00:11.2634357Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 106952 2023-01-11T23:00:11.2634731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2634908Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2635289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2635478Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2635839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2635993Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2636369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2636558Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2636914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2637084Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2637458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2637644Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2638011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2638180Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2638534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2638770Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2639028Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7u86xa26 2023-01-11T23:00:11.2639297Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7u86xa26/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2639555Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv841nuq1 2023-01-11T23:00:11.2639821Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv841nuq1/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2640073Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2vh1qse1 2023-01-11T23:00:11.2640335Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2vh1qse1/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2640567Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8p6ew0dz 2023-01-11T23:00:11.2640834Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8p6ew0dz/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2641063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2641288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2641570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2641801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2641947Z fi_getinfo: -61 2023-01-11T23:00:11.2642083Z fi_getinfo: -61 2023-01-11T23:00:11.2642203Z fi_getinfo: -61 2023-01-11T23:00:11.2642339Z fi_getinfo: -61 2023-01-11T23:00:11.2642437Z ok (10.026s) 2023-01-11T23:00:11.2642457Z 2023-01-11T23:00:11.2642719Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2642836Z Ran 1 test in 10.026s 2023-01-11T23:00:11.2642855Z 2023-01-11T23:00:11.2642946Z OK 2023-01-11T23:00:11.2642964Z 2023-01-11T23:00:11.2643087Z Generating XML reports... 2023-01-11T23:00:11.2643634Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225938.xml 2023-01-11T23:00:11.2643987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2644165Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2644543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2644732Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2644989Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi43k3h9_ 2023-01-11T23:00:11.2645254Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi43k3h9_/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2645277Z 2023-01-11T23:00:11.2645384Z Running tests... 2023-01-11T23:00:11.2645646Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2646004Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2646311Z test_device_maps_backward_pass (__main__.TensorPipeTensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2646534Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107596 2023-01-11T23:00:11.2646752Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107597 2023-01-11T23:00:11.2646969Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 107598 2023-01-11T23:00:11.2647180Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 107599 2023-01-11T23:00:11.2647609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2647784Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2648164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2648338Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2648706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2648876Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2649247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2649433Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2649794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2649968Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2650543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2650742Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2651166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2651346Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2651719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2651904Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2652159Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4n6rr03t 2023-01-11T23:00:11.2652430Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4n6rr03t/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2652687Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc803xs_z 2023-01-11T23:00:11.2652957Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc803xs_z/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2653194Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj6zivdu5 2023-01-11T23:00:11.2653464Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj6zivdu5/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2653714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp37rlzw4z 2023-01-11T23:00:11.2653976Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp37rlzw4z/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2654204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2654434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2654662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2654922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2655072Z skip: Need at least 4 CUDA devices (4.391s) 2023-01-11T23:00:11.2655092Z 2023-01-11T23:00:11.2655345Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2655456Z Ran 1 test in 4.391s 2023-01-11T23:00:11.2655474Z 2023-01-11T23:00:11.2655580Z OK (skipped=1) 2023-01-11T23:00:11.2655600Z 2023-01-11T23:00:11.2655723Z Generating XML reports... 2023-01-11T23:00:11.2656292Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20230111225951.xml 2023-01-11T23:00:11.2656662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2656911Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2657296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2657486Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2657723Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0eoux32x 2023-01-11T23:00:11.2657995Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0eoux32x/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2658014Z 2023-01-11T23:00:11.2658122Z Running tests... 2023-01-11T23:00:11.2658383Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2658740Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2659062Z test_dist_autograd_sync_streams (__main__.TensorPipeTensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2659286Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107767 2023-01-11T23:00:11.2659501Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107768 2023-01-11T23:00:11.2659701Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 107769 2023-01-11T23:00:11.2659955Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 107770 2023-01-11T23:00:11.2660338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2660512Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2660893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2661083Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2661451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2661625Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2662000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2662172Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2662535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2662707Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2663078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2663264Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2663631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2663806Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2664180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2664347Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2664605Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4_a_0j45 2023-01-11T23:00:11.2664869Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4_a_0j45/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2665125Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz4c7yjv_ 2023-01-11T23:00:11.2665392Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz4c7yjv_/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2665643Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzgop0xl2 2023-01-11T23:00:11.2665961Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzgop0xl2/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2666213Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy3ch0u50 2023-01-11T23:00:11.2666476Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy3ch0u50/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2666692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2666920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2667145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2667368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2667515Z skip: Need at least 4 CUDA devices (4.464s) 2023-01-11T23:00:11.2667534Z 2023-01-11T23:00:11.2667808Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2667921Z Ran 1 test in 4.464s 2023-01-11T23:00:11.2667940Z 2023-01-11T23:00:11.2668046Z OK (skipped=1) 2023-01-11T23:00:11.2668064Z 2023-01-11T23:00:11.2668170Z Generating XML reports... 2023-01-11T23:00:11.2668741Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20230111225958.xml 2023-01-11T23:00:11.2669160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2669339Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2669722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2669912Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2670166Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplri8s_nz 2023-01-11T23:00:11.2670440Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplri8s_nz/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2670458Z 2023-01-11T23:00:11.2670569Z Running tests... 2023-01-11T23:00:11.2670813Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2671178Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2023-01-11T23:00:11.2671509Z test_gradients_synchronizations (__main__.TensorPipeTensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init 2023-01-11T23:00:11.2671731Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107938 2023-01-11T23:00:11.2671949Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107939 2023-01-11T23:00:11.2672164Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 107940 2023-01-11T23:00:11.2672379Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 107941 2023-01-11T23:00:11.2672755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2672930Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2673305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2673490Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2673856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2674030Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2674407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2674594Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2675008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2675178Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2675533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2675718Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2676086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests 2023-01-11T23:00:11.2676257Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2023-01-11T23:00:11.2676632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests 2023-01-11T23:00:11.2676816Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2023-01-11T23:00:11.2677074Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnjku80hm 2023-01-11T23:00:11.2677343Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnjku80hm/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2677598Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphc_ktwt5 2023-01-11T23:00:11.2677848Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphc_ktwt5/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2678146Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk4y1rtpc 2023-01-11T23:00:11.2678421Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk4y1rtpc/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2678673Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe8lywcfi 2023-01-11T23:00:11.2678937Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe8lywcfi/_remote_module_non_scriptable.py 2023-01-11T23:00:11.2679168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2023-01-11T23:00:11.2679399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2023-01-11T23:00:11.2679626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2023-01-11T23:00:11.2679834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2023-01-11T23:00:11.2679983Z skip: Need at least 4 CUDA devices (4.414s) 2023-01-11T23:00:11.2680006Z 2023-01-11T23:00:11.2680274Z ---------------------------------------------------------------------- 2023-01-11T23:00:11.2680387Z Ran 1 test in 4.415s 2023-01-11T23:00:11.2680406Z 2023-01-11T23:00:11.2680514Z OK (skipped=1) 2023-01-11T23:00:11.2680533Z 2023-01-11T23:00:11.2680655Z Generating XML reports... 2023-01-11T23:00:11.2681221Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20230111230005.xml 2023-01-11T23:00:11.2681244Z 2023-01-11T23:00:11.2681672Z ##[endgroup] 2023-01-11T23:00:11.2682178Z FINISHED PRINTING LOG FILE of distributed/rpc/cuda/test_tensorpipe_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-cuda-test_tensorpipe_agent_udxpg47p) 2023-01-11T23:00:11.2682215Z 2023-01-11T23:00:11.4154511Z 2023-01-11T23:00:11.4155099Z real 107m10.286s 2023-01-11T23:00:11.4155201Z user 185m2.023s 2023-01-11T23:00:11.4155318Z sys 94m53.116s 2023-01-11T23:00:11.4155459Z + assert_git_not_dirty 2023-01-11T23:00:11.4155849Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 != *rocm* ]] 2023-01-11T23:00:11.4156089Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 != *xla* ]] 2023-01-11T23:00:11.4158404Z ++ git status --porcelain 2023-01-11T23:00:13.0243441Z + git_status= 2023-01-11T23:00:13.0243850Z + [[ -n '' ]] 2023-01-11T23:00:13.0244267Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *cuda* ]] 2023-01-11T23:00:13.0244562Z + [[ 1 == 1 ]] 2023-01-11T23:00:13.0244861Z + echo 'Testing distributed C++ tests' 2023-01-11T23:00:13.0245364Z Testing distributed C++ tests 2023-01-11T23:00:13.0246760Z + ln -sf /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_linalg.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorchbind_test.so /opt/conda/lib/python3.10/site-packages/torch/bin 2023-01-11T23:00:13.0259129Z + ln -sf /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so /opt/conda/lib/python3.10/site-packages/torch/lib/libc10d_cuda_test.so /opt/conda/lib/python3.10/site-packages/torch/bin 2023-01-11T23:00:13.0269298Z + TEST_REPORTS_DIR=test/test-reports/cpp-distributed/test_distributed 2023-01-11T23:00:13.0269804Z + mkdir -p test/test-reports/cpp-distributed/test_distributed 2023-01-11T23:00:13.0284153Z + /opt/conda/lib/python3.10/site-packages/torch/bin/FileStoreTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/FileStoreTest.xml 2023-01-11T23:00:13.3620920Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc 2023-01-11T23:00:13.3621812Z [==========] Running 4 tests from 1 test suite. 2023-01-11T23:00:13.3622440Z [----------] Global test environment set-up. 2023-01-11T23:00:13.3622867Z [----------] 4 tests from FileStoreTest 2023-01-11T23:00:13.3623263Z [ RUN ] FileStoreTest.testGetAndSet 2023-01-11T23:00:13.3626934Z [ OK ] FileStoreTest.testGetAndSet (0 ms) 2023-01-11T23:00:13.3627407Z [ RUN ] FileStoreTest.testGetAndSetWithPrefix 2023-01-11T23:00:13.3632471Z [ OK ] FileStoreTest.testGetAndSetWithPrefix (0 ms) 2023-01-11T23:00:13.3632935Z [ RUN ] FileStoreTest.testStressStore 2023-01-11T23:00:13.3836042Z [ OK ] FileStoreTest.testStressStore (20 ms) 2023-01-11T23:00:13.3836520Z [ RUN ] FileStoreTest.testStressStoreWithPrefix 2023-01-11T23:00:13.4036973Z [ OK ] FileStoreTest.testStressStoreWithPrefix (20 ms) 2023-01-11T23:00:13.4037730Z [----------] 4 tests from FileStoreTest (41 ms total) 2023-01-11T23:00:13.4037960Z 2023-01-11T23:00:13.4038202Z [----------] Global test environment tear-down 2023-01-11T23:00:13.4039026Z [==========] 4 tests from 1 test suite ran. (41 ms total) 2023-01-11T23:00:13.4039631Z [ PASSED ] 4 tests. 2023-01-11T23:00:13.4726647Z + /opt/conda/lib/python3.10/site-packages/torch/bin/HashStoreTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/HashStoreTest.xml 2023-01-11T23:00:13.8063720Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc 2023-01-11T23:00:13.8064652Z [==========] Running 4 tests from 1 test suite. 2023-01-11T23:00:13.8065056Z [----------] Global test environment set-up. 2023-01-11T23:00:13.8065664Z [----------] 4 tests from HashStoreTest 2023-01-11T23:00:13.8066105Z [ RUN ] HashStoreTest.testGetAndSet 2023-01-11T23:00:13.9069090Z [ OK ] HashStoreTest.testGetAndSet (100 ms) 2023-01-11T23:00:13.9069573Z [ RUN ] HashStoreTest.testGetAndSetWithPrefix 2023-01-11T23:00:14.0071406Z [ OK ] HashStoreTest.testGetAndSetWithPrefix (100 ms) 2023-01-11T23:00:14.0072307Z [ RUN ] HashStoreTest.testStressStore 2023-01-11T23:00:14.0077058Z [ OK ] HashStoreTest.testStressStore (0 ms) 2023-01-11T23:00:14.0077522Z [ RUN ] HashStoreTest.testStressStoreWithPrefix 2023-01-11T23:00:14.0083173Z [ OK ] HashStoreTest.testStressStoreWithPrefix (0 ms) 2023-01-11T23:00:14.0083687Z [----------] 4 tests from HashStoreTest (202 ms total) 2023-01-11T23:00:14.0084098Z 2023-01-11T23:00:14.0084329Z [----------] Global test environment tear-down 2023-01-11T23:00:14.0086042Z [==========] 4 tests from 1 test suite ran. (202 ms total) 2023-01-11T23:00:14.0086415Z [ PASSED ] 4 tests. 2023-01-11T23:00:14.0780706Z + /opt/conda/lib/python3.10/site-packages/torch/bin/TCPStoreTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/TCPStoreTest.xml 2023-01-11T23:00:14.4199732Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc 2023-01-11T23:00:14.4200373Z [==========] Running 11 tests from 1 test suite. 2023-01-11T23:00:14.4200765Z [----------] Global test environment set-up. 2023-01-11T23:00:14.4201152Z [----------] 11 tests from TCPStoreTest 2023-01-11T23:00:14.4201520Z [ RUN ] TCPStoreTest.testHelper 2023-01-11T23:00:15.3780920Z [ OK ] TCPStoreTest.testHelper (958 ms) 2023-01-11T23:00:15.3781395Z [ RUN ] TCPStoreTest.testHelperPrefix 2023-01-11T23:00:16.3539957Z [ OK ] TCPStoreTest.testHelperPrefix (975 ms) 2023-01-11T23:00:16.3540429Z [ RUN ] TCPStoreTest.testWatchKeyCallback 2023-01-11T23:00:16.3678525Z [ OK ] TCPStoreTest.testWatchKeyCallback (13 ms) 2023-01-11T23:00:16.3679049Z [ RUN ] TCPStoreTest.testWatchKeyCallbackWithPrefix 2023-01-11T23:00:16.3820715Z [ OK ] TCPStoreTest.testWatchKeyCallbackWithPrefix (14 ms) 2023-01-11T23:00:16.3821247Z [ RUN ] TCPStoreTest.testKeyEmptyUpdate 2023-01-11T23:00:16.5899707Z [ OK ] TCPStoreTest.testKeyEmptyUpdate (207 ms) 2023-01-11T23:00:16.5900162Z [ RUN ] TCPStoreTest.testKeyUpdate 2023-01-11T23:00:16.5905609Z [ OK ] TCPStoreTest.testKeyUpdate (0 ms) 2023-01-11T23:00:16.5906029Z [ RUN ] TCPStoreTest.testKeyCreate 2023-01-11T23:00:16.5910675Z [ OK ] TCPStoreTest.testKeyCreate (0 ms) 2023-01-11T23:00:16.5911071Z [ RUN ] TCPStoreTest.testKeyAdd 2023-01-11T23:00:16.5915765Z [ OK ] TCPStoreTest.testKeyAdd (0 ms) 2023-01-11T23:00:16.5916178Z [ RUN ] TCPStoreTest.testKeyDelete 2023-01-11T23:00:16.7976516Z [ OK ] TCPStoreTest.testKeyDelete (206 ms) 2023-01-11T23:00:16.7976961Z [ RUN ] TCPStoreTest.testCleanShutdown 2023-01-11T23:00:16.7983521Z [ OK ] TCPStoreTest.testCleanShutdown (0 ms) 2023-01-11T23:00:16.7983964Z [ RUN ] TCPStoreTest.testMultiTenantStores 2023-01-11T23:00:16.7996292Z [ OK ] TCPStoreTest.testMultiTenantStores (1 ms) 2023-01-11T23:00:16.7997066Z [----------] 11 tests from TCPStoreTest (2379 ms total) 2023-01-11T23:00:16.7997283Z 2023-01-11T23:00:16.7997508Z [----------] Global test environment tear-down 2023-01-11T23:00:16.7999594Z [==========] 11 tests from 1 test suite ran. (2379 ms total) 2023-01-11T23:00:16.8000048Z [ PASSED ] 11 tests. 2023-01-11T23:00:16.8742783Z ++ command -v mpiexec 2023-01-11T23:00:16.8745120Z + MPIEXEC=/usr/bin/mpiexec 2023-01-11T23:00:16.8745440Z + [[ -n /usr/bin/mpiexec ]] 2023-01-11T23:00:16.8745937Z + [[ -z true ]] 2023-01-11T23:00:16.8746638Z + /opt/conda/lib/python3.10/site-packages/torch/bin/ProcessGroupGlooTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/ProcessGroupGlooTest.xml 2023-01-11T23:00:17.2103409Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc 2023-01-11T23:00:17.2104002Z [==========] Running 12 tests from 1 test suite. 2023-01-11T23:00:17.2104408Z [----------] Global test environment set-up. 2023-01-11T23:00:17.2104838Z [----------] 12 tests from ProcessGroupGlooTest 2023-01-11T23:00:17.2105287Z [ RUN ] ProcessGroupGlooTest.testSIGSTOPException 2023-01-11T23:00:18.2663257Z [ OK ] ProcessGroupGlooTest.testSIGSTOPException (1055 ms) 2023-01-11T23:00:18.2663990Z [ RUN ] ProcessGroupGlooTest.testSIGKILLException 2023-01-11T23:00:18.2994952Z [ OK ] ProcessGroupGlooTest.testSIGKILLException (33 ms) 2023-01-11T23:00:18.2995712Z [ RUN ] ProcessGroupGlooTest.testAllReduceCPU 2023-01-11T23:00:18.5978896Z [ OK ] ProcessGroupGlooTest.testAllReduceCPU (298 ms) 2023-01-11T23:00:18.5979428Z [ RUN ] ProcessGroupGlooTest.testBroadcastCPU 2023-01-11T23:00:18.6506482Z [ OK ] ProcessGroupGlooTest.testBroadcastCPU (52 ms) 2023-01-11T23:00:18.6506978Z [ RUN ] ProcessGroupGlooTest.testAllToAllCPU 2023-01-11T23:00:18.8041658Z [ OK ] ProcessGroupGlooTest.testAllToAllCPU (153 ms) 2023-01-11T23:00:18.8042149Z [ RUN ] ProcessGroupGlooTest.testBarrier 2023-01-11T23:00:18.8458645Z [ OK ] ProcessGroupGlooTest.testBarrier (41 ms) 2023-01-11T23:00:18.8459134Z [ RUN ] ProcessGroupGlooTest.testMonitoredBarrier 2023-01-11T23:00:19.8665680Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 1000 ms 2023-01-11T23:00:19.8868534Z [ OK ] ProcessGroupGlooTest.testMonitoredBarrier (1040 ms) 2023-01-11T23:00:19.8869088Z [ RUN ] ProcessGroupGlooTest.testSequenceNumInit 2023-01-11T23:00:20.0190207Z [ OK ] ProcessGroupGlooTest.testSequenceNumInit (132 ms) 2023-01-11T23:00:20.0190888Z [ RUN ] ProcessGroupGlooTest.testSend 2023-01-11T23:00:20.0709515Z [ OK ] ProcessGroupGlooTest.testSend (51 ms) 2023-01-11T23:00:20.0709967Z [ RUN ] ProcessGroupGlooTest.testRecv 2023-01-11T23:00:20.1127629Z [ OK ] ProcessGroupGlooTest.testRecv (41 ms) 2023-01-11T23:00:20.1128093Z [ RUN ] ProcessGroupGlooTest.testStoreSetGet 2023-01-11T23:00:20.1537847Z [ OK ] ProcessGroupGlooTest.testStoreSetGet (40 ms) 2023-01-11T23:00:20.1538340Z [ RUN ] ProcessGroupGlooTest.testWaitDelay 2023-01-11T23:00:20.2956929Z [ OK ] ProcessGroupGlooTest.testWaitDelay (141 ms) 2023-01-11T23:00:20.2957955Z [----------] 12 tests from ProcessGroupGlooTest (3085 ms total) 2023-01-11T23:00:20.2958465Z 2023-01-11T23:00:20.2958945Z [----------] Global test environment tear-down 2023-01-11T23:00:20.2960395Z [==========] 12 tests from 1 test suite ran. (3085 ms total) 2023-01-11T23:00:20.2961115Z [ PASSED ] 12 tests. 2023-01-11T23:00:20.3823575Z + /opt/conda/lib/python3.10/site-packages/torch/bin/ProcessGroupNCCLTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/ProcessGroupNCCLTest.xml 2023-01-11T23:00:20.8165623Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc 2023-01-11T23:00:20.8166249Z [==========] Running 11 tests from 1 test suite. 2023-01-11T23:00:20.8166650Z [----------] Global test environment set-up. 2023-01-11T23:00:20.8167228Z [----------] 11 tests from ProcessGroupNCCLTest 2023-01-11T23:00:20.8167718Z [ RUN ] ProcessGroupNCCLTest.testAllreduce 2023-01-11T23:00:25.3406081Z [ OK ] ProcessGroupNCCLTest.testAllreduce (4523 ms) 2023-01-11T23:00:25.3406610Z [ RUN ] ProcessGroupNCCLTest.testBroadcast 2023-01-11T23:00:28.8015305Z [ OK ] ProcessGroupNCCLTest.testBroadcast (3460 ms) 2023-01-11T23:00:28.8015848Z [ RUN ] ProcessGroupNCCLTest.testReduce 2023-01-11T23:00:32.2621289Z [ OK ] ProcessGroupNCCLTest.testReduce (3460 ms) 2023-01-11T23:00:32.2621831Z [ RUN ] ProcessGroupNCCLTest.testAllgather 2023-01-11T23:00:34.0284912Z [ OK ] ProcessGroupNCCLTest.testAllgather (1766 ms) 2023-01-11T23:00:34.0285450Z [ RUN ] ProcessGroupNCCLTest.testAllgatherBase 2023-01-11T23:00:35.7787973Z [ OK ] ProcessGroupNCCLTest.testAllgatherBase (1750 ms) 2023-01-11T23:00:35.7788774Z [ RUN ] ProcessGroupNCCLTest.testReduceScatter 2023-01-11T23:00:37.5390722Z [ OK ] ProcessGroupNCCLTest.testReduceScatter (1760 ms) 2023-01-11T23:00:37.5391284Z [ RUN ] ProcessGroupNCCLTest.testSequenceNumInit 2023-01-11T23:00:37.6402085Z [ OK ] ProcessGroupNCCLTest.testSequenceNumInit (101 ms) 2023-01-11T23:00:37.6402786Z [ RUN ] ProcessGroupNCCLTest.testProcessGroupNCCLHealthCheckFailTimeout 2023-01-11T23:00:40.6449336Z [ OK ] ProcessGroupNCCLTest.testProcessGroupNCCLHealthCheckFailTimeout (3004 ms) 2023-01-11T23:00:40.6450180Z [ RUN ] ProcessGroupNCCLTest.testProcessGroupNCCLHealthCheckFailException 2023-01-11T23:00:43.6465008Z [ OK ] ProcessGroupNCCLTest.testProcessGroupNCCLHealthCheckFailException (3001 ms) 2023-01-11T23:00:43.6465679Z [ RUN ] ProcessGroupNCCLTest.testReduceScatterBase 2023-01-11T23:00:45.7831151Z [ OK ] ProcessGroupNCCLTest.testReduceScatterBase (2136 ms) 2023-01-11T23:00:45.7831695Z [ RUN ] ProcessGroupNCCLTest.testBackendName 2023-01-11T23:00:45.8250824Z [ OK ] ProcessGroupNCCLTest.testBackendName (41 ms) 2023-01-11T23:00:45.8251408Z [----------] 11 tests from ProcessGroupNCCLTest (25008 ms total) 2023-01-11T23:00:45.8251665Z 2023-01-11T23:00:45.8251892Z [----------] Global test environment tear-down 2023-01-11T23:00:45.8254532Z [==========] 11 tests from 1 test suite ran. (25008 ms total) 2023-01-11T23:00:45.8254968Z [ PASSED ] 11 tests. 2023-01-11T23:00:46.0516762Z + /opt/conda/lib/python3.10/site-packages/torch/bin/ProcessGroupNCCLErrorsTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/ProcessGroupNCCLErrorsTest.xml 2023-01-11T23:00:46.4793451Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc 2023-01-11T23:00:46.4794217Z [==========] Running 3 tests from 1 test suite. 2023-01-11T23:00:46.4794625Z [----------] Global test environment set-up. 2023-01-11T23:00:46.4795070Z [----------] 3 tests from ProcessGroupNCCLErrorsTest 2023-01-11T23:00:46.4795697Z [ RUN ] ProcessGroupNCCLErrorsTest.testNCCLErrorsBlocking 2023-01-11T23:00:48.8980734Z [ OK ] ProcessGroupNCCLErrorsTest.testNCCLErrorsBlocking (2418 ms) 2023-01-11T23:00:48.8981430Z [ RUN ] ProcessGroupNCCLErrorsTest.testNCCLTimedoutErrorsBlocking 2023-01-11T23:00:52.0044697Z [ OK ] ProcessGroupNCCLErrorsTest.testNCCLTimedoutErrorsBlocking (3106 ms) 2023-01-11T23:00:52.0045363Z [ RUN ] ProcessGroupNCCLErrorsTest.testNCCLErrorsNonBlocking 2023-01-11T23:00:52.1050400Z [ OK ] ProcessGroupNCCLErrorsTest.testNCCLErrorsNonBlocking (100 ms) 2023-01-11T23:00:52.1051300Z [----------] 3 tests from ProcessGroupNCCLErrorsTest (5625 ms total) 2023-01-11T23:00:52.1051565Z 2023-01-11T23:00:52.1051821Z [----------] Global test environment tear-down 2023-01-11T23:00:52.1052216Z [==========] 3 tests from 1 test suite ran. (5625 ms total) 2023-01-11T23:00:52.1052569Z [ PASSED ] 3 tests. 2023-01-11T23:00:52.3270813Z + [[ 1 == 1 ]] 2023-01-11T23:00:52.3271105Z + test_rpc 2023-01-11T23:00:52.3271578Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 != *rocm* ]] 2023-01-11T23:00:52.3271957Z + echo 'Testing RPC C++ tests' 2023-01-11T23:00:52.3272212Z Testing RPC C++ tests 2023-01-11T23:00:52.3273664Z + ln -sf /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_linalg.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorchbind_test.so /opt/conda/lib/python3.10/site-packages/torch/bin 2023-01-11T23:00:52.3288547Z + ln -sf /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so /opt/conda/lib/python3.10/site-packages/torch/lib/libc10d_cuda_test.so /opt/conda/lib/python3.10/site-packages/torch/bin 2023-01-11T23:00:52.3300792Z + ln -sf '/opt/conda/lib/python3.10/site-packages/torch/lib/libtbb*' /opt/conda/lib/python3.10/site-packages/torch/bin 2023-01-11T23:00:52.3310290Z + TEST_REPORTS_DIR=test/test-reports/cpp-rpc/test_rpc 2023-01-11T23:00:52.3310702Z + mkdir -p test/test-reports/cpp-rpc/test_rpc 2023-01-11T23:00:52.3323915Z + /opt/conda/lib/python3.10/site-packages/torch/bin/test_cpp_rpc --gtest_output=xml:test/test-reports/cpp-rpc/test_rpc/test_cpp_rpc.xml 2023-01-11T23:00:52.7909350Z [==========] Running 8 tests from 3 test suites. 2023-01-11T23:00:52.7909825Z [----------] Global test environment set-up. 2023-01-11T23:00:52.7910211Z [----------] 4 tests from WireSerialize 2023-01-11T23:00:52.7910704Z [ RUN ] WireSerialize.Base 2023-01-11T23:00:52.7924126Z [ OK ] WireSerialize.Base (1 ms) 2023-01-11T23:00:52.7924547Z [ RUN ] WireSerialize.RecopySparseTensors 2023-01-11T23:00:52.8030186Z [ OK ] WireSerialize.RecopySparseTensors (10 ms) 2023-01-11T23:00:52.8030662Z [ RUN ] WireSerialize.CloneSparseTensors 2023-01-11T23:00:52.8131843Z [ OK ] WireSerialize.CloneSparseTensors (10 ms) 2023-01-11T23:00:52.8132278Z [ RUN ] WireSerialize.Errors 2023-01-11T23:00:52.8159306Z [ OK ] WireSerialize.Errors (2 ms) 2023-01-11T23:00:52.8159754Z [----------] 4 tests from WireSerialize (25 ms total) 2023-01-11T23:00:52.8159974Z 2023-01-11T23:00:52.8160195Z [----------] 1 test from TestE2ETensorPipe 2023-01-11T23:00:52.8160603Z [ RUN ] TestE2ETensorPipe.TestTrainingLoop 2023-01-11T23:00:53.5540509Z [W tensorpipe_agent.cpp:725] RPC agent for worker encountered error when reading incoming request from worker: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2023-01-11T23:00:53.5560094Z [ OK ] TestE2ETensorPipe.TestTrainingLoop (739 ms) 2023-01-11T23:00:53.5560611Z [----------] 1 test from TestE2ETensorPipe (740 ms total) 2023-01-11T23:00:53.5560840Z 2023-01-11T23:00:53.5561074Z [----------] 3 tests from TensorpipeSerialize 2023-01-11T23:00:53.5561484Z [ RUN ] TensorpipeSerialize.Base 2023-01-11T23:00:53.5561885Z [ OK ] TensorpipeSerialize.Base (0 ms) 2023-01-11T23:00:53.5562346Z [ RUN ] TensorpipeSerialize.RecopySparseTensors 2023-01-11T23:00:53.5667583Z [ OK ] TensorpipeSerialize.RecopySparseTensors (10 ms) 2023-01-11T23:00:53.5668516Z [ RUN ] TensorpipeSerialize.NoDeleterTensors 2023-01-11T23:00:53.5669427Z [ OK ] TensorpipeSerialize.NoDeleterTensors (0 ms) 2023-01-11T23:00:53.5670343Z [----------] 3 tests from TensorpipeSerialize (10 ms total) 2023-01-11T23:00:53.5670771Z 2023-01-11T23:00:53.5671188Z [----------] Global test environment tear-down 2023-01-11T23:00:53.5671981Z [==========] 8 tests from 3 test suites ran. (776 ms total) 2023-01-11T23:00:53.5672604Z [ PASSED ] 8 tests. 2023-01-11T23:00:53.5672905Z 2023-01-11T23:00:53.5673210Z  YOU HAVE 1 DISABLED TEST 2023-01-11T23:00:53.5673515Z 2023-01-11T23:00:53.6784793Z  2023-01-11T23:00:53.6815064Z ##[group]Run cat test/**/*.log || true 2023-01-11T23:00:53.6815402Z cat test/**/*.log || true 2023-01-11T23:00:53.6828962Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-01-11T23:00:53.6829264Z env: 2023-01-11T23:00:53.6829512Z GIT_DEFAULT_BRANCH: master 2023-01-11T23:00:53.6829766Z GPU_FLAG: --gpus all 2023-01-11T23:00:53.6830112Z DOCKER_CONTAINER_ID: 76a1849f93d45a34422360460369a49f22eebafa5f0784e84d188aa13b21145b 2023-01-11T23:00:53.6830592Z ##[endgroup] 2023-01-11T23:00:53.6876384Z cat: test/**/*.log: No such file or directory 2023-01-11T23:00:53.6907165Z Prepare all required actions 2023-01-11T23:00:53.6907588Z Getting action download info 2023-01-11T23:00:53.8727797Z ##[group]Run ./.github/actions/get-workflow-job-id 2023-01-11T23:00:53.8728072Z with: 2023-01-11T23:00:53.8728537Z github-token: *** 2023-01-11T23:00:53.8728780Z env: 2023-01-11T23:00:53.8729001Z GIT_DEFAULT_BRANCH: master 2023-01-11T23:00:53.8729266Z GPU_FLAG: --gpus all 2023-01-11T23:00:53.8729608Z DOCKER_CONTAINER_ID: 76a1849f93d45a34422360460369a49f22eebafa5f0784e84d188aa13b21145b 2023-01-11T23:00:53.8729936Z ##[endgroup] 2023-01-11T23:00:53.8763996Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2023-01-11T23:00:53.8764310Z with: 2023-01-11T23:00:53.8764528Z shell: bash 2023-01-11T23:00:53.8764752Z timeout_minutes: 10 2023-01-11T23:00:53.8764998Z max_attempts: 5 2023-01-11T23:00:53.8765244Z retry_wait_seconds: 30 2023-01-11T23:00:53.8765774Z command: set -eux python3 -m pip install requests==2.26.0 GHA_WORKFLOW_JOB_ID=$(python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}") echo "job-id=${GHA_WORKFLOW_JOB_ID}" >> "${GITHUB_OUTPUT}" 2023-01-11T23:00:53.8766281Z polling_interval_seconds: 1 2023-01-11T23:00:53.8766530Z warning_on_retry: true 2023-01-11T23:00:53.8766791Z continue_on_error: false 2023-01-11T23:00:53.8767029Z env: 2023-01-11T23:00:53.8767245Z GIT_DEFAULT_BRANCH: master 2023-01-11T23:00:53.8767508Z GPU_FLAG: --gpus all 2023-01-11T23:00:53.8767845Z DOCKER_CONTAINER_ID: 76a1849f93d45a34422360460369a49f22eebafa5f0784e84d188aa13b21145b 2023-01-11T23:00:53.8768320Z GITHUB_TOKEN: *** 2023-01-11T23:00:53.8768548Z ##[endgroup] 2023-01-11T23:00:53.9438359Z + python3 -m pip install requests==2.26.0 2023-01-11T23:00:54.2257803Z Defaulting to user installation because normal site-packages is not writeable 2023-01-11T23:00:54.3633977Z Collecting requests==2.26.0 2023-01-11T23:00:54.3841868Z Downloading requests-2.26.0-py2.py3-none-any.whl (62 kB) 2023-01-11T23:00:54.4664874Z Collecting certifi>=2017.4.17 2023-01-11T23:00:54.4714957Z Downloading certifi-2022.12.7-py3-none-any.whl (155 kB) 2023-01-11T23:00:54.5791974Z Collecting urllib3<1.27,>=1.21.1 2023-01-11T23:00:54.5961044Z Downloading urllib3-1.26.14-py2.py3-none-any.whl (140 kB) 2023-01-11T23:00:54.6702398Z Collecting idna<4,>=2.5; python_version >= "3" 2023-01-11T23:00:54.6751946Z Downloading idna-3.4-py3-none-any.whl (61 kB) 2023-01-11T23:00:54.8380886Z Collecting charset-normalizer~=2.0.0; python_version >= "3" 2023-01-11T23:00:54.8428856Z Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2023-01-11T23:00:54.9277667Z Installing collected packages: certifi, urllib3, idna, charset-normalizer, requests 2023-01-11T23:00:55.1362050Z WARNING: The script normalizer is installed in '/home/ec2-user/.local/bin' which is not on PATH. 2023-01-11T23:00:55.1362707Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2023-01-11T23:00:55.1846628Z Successfully installed certifi-2022.12.7 charset-normalizer-2.0.12 idna-3.4 requests-2.26.0 urllib3-1.26.14 2023-01-11T23:00:55.2352727Z ++ python3 .github/scripts/get_workflow_job_id.py 3896099317 i-011ccd2732b3aac12 2023-01-11T23:00:56.8701725Z + GHA_WORKFLOW_JOB_ID=10589291239 2023-01-11T23:00:56.8702416Z + echo job-id=10589291239 2023-01-11T23:00:56.9427376Z Command completed after 1 attempt(s). 2023-01-11T23:00:56.9568245Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2023-01-11T23:00:56.9568594Z kill "$MONITOR_SCRIPT_PID" 2023-01-11T23:00:56.9582550Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-01-11T23:00:56.9582852Z env: 2023-01-11T23:00:56.9583093Z GIT_DEFAULT_BRANCH: master 2023-01-11T23:00:56.9583339Z GPU_FLAG: --gpus all 2023-01-11T23:00:56.9583686Z DOCKER_CONTAINER_ID: 76a1849f93d45a34422360460369a49f22eebafa5f0784e84d188aa13b21145b 2023-01-11T23:00:56.9584030Z MONITOR_SCRIPT_PID: 60103 2023-01-11T23:00:56.9584388Z ##[endgroup] 2023-01-11T23:00:56.9687387Z Prepare all required actions 2023-01-11T23:00:56.9687749Z Getting action download info 2023-01-11T23:00:57.1327622Z Download action repository 'actions/upload-artifact@v3' (SHA:0b7f8abb1508181956e8e162db84b466c27e18ce) 2023-01-11T23:00:57.2877779Z ##[group]Run ./.github/actions/upload-test-artifacts 2023-01-11T23:00:57.2878057Z with: 2023-01-11T23:00:57.2878405Z file-suffix: test-distributed-1-3-linux.8xlarge.nvidia.gpu_10589291239 2023-01-11T23:00:57.2878745Z env: 2023-01-11T23:00:57.2878962Z GIT_DEFAULT_BRANCH: master 2023-01-11T23:00:57.2879222Z GPU_FLAG: --gpus all 2023-01-11T23:00:57.2879561Z DOCKER_CONTAINER_ID: 76a1849f93d45a34422360460369a49f22eebafa5f0784e84d188aa13b21145b 2023-01-11T23:00:57.2879869Z ##[endgroup] 2023-01-11T23:00:57.2907816Z ##[group]Run # Remove any previous test jsons if they exist 2023-01-11T23:00:57.2908169Z # Remove any previous test jsons if they exist 2023-01-11T23:00:57.2908494Z rm -f test-jsons-*.zip 2023-01-11T23:00:57.2908867Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2023-01-11T23:00:57.2920894Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-01-11T23:00:57.2921186Z env: 2023-01-11T23:00:57.2921428Z GIT_DEFAULT_BRANCH: master 2023-01-11T23:00:57.2921675Z GPU_FLAG: --gpus all 2023-01-11T23:00:57.2922020Z DOCKER_CONTAINER_ID: 76a1849f93d45a34422360460369a49f22eebafa5f0784e84d188aa13b21145b 2023-01-11T23:00:57.2922472Z FILE_SUFFIX: test-distributed-1-3-linux.8xlarge.nvidia.gpu_10589291239 2023-01-11T23:00:57.2922805Z ##[endgroup] 2023-01-11T23:00:57.3076306Z adding: test/allowlist_for_publicAPI.json (deflated 78%) 2023-01-11T23:00:57.3112912Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2023-01-11T23:00:57.3120225Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2023-01-11T23:00:57.3121543Z adding: test/.pytorch-slow-tests.json (deflated 74%) 2023-01-11T23:00:57.3127537Z adding: test/.pytorch-disabled-tests.json (deflated 84%) 2023-01-11T23:00:57.3153151Z ##[group]Run # Remove any previous test reports if they exist 2023-01-11T23:00:57.3153515Z # Remove any previous test reports if they exist 2023-01-11T23:00:57.3153835Z rm -f test-reports-*.zip 2023-01-11T23:00:57.3154185Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2023-01-11T23:00:57.3166305Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-01-11T23:00:57.3166599Z env: 2023-01-11T23:00:57.3166840Z GIT_DEFAULT_BRANCH: master 2023-01-11T23:00:57.3167086Z GPU_FLAG: --gpus all 2023-01-11T23:00:57.3167426Z DOCKER_CONTAINER_ID: 76a1849f93d45a34422360460369a49f22eebafa5f0784e84d188aa13b21145b 2023-01-11T23:00:57.3167873Z FILE_SUFFIX: test-distributed-1-3-linux.8xlarge.nvidia.gpu_10589291239 2023-01-11T23:00:57.3168223Z ##[endgroup] 2023-01-11T23:00:57.3320954Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_ignored_modules/TEST-TestFSDPIgnoredModules-20230111211307.xml (deflated 76%) 2023-01-11T23:00:57.3322194Z adding: test/test-reports/python-unittest/distributed._composable.test_checkpoint/TEST-TestCheckpoint-20230111221447.xml (deflated 74%) 2023-01-11T23:00:57.3323345Z adding: test/test-reports/python-unittest/distributed._tools.test_memory_tracker/TEST-TestMemoryTracker-20230111221451.xml (deflated 61%) 2023-01-11T23:00:57.3324462Z adding: test/test-reports/python-unittest/distributed.nn.jit.test_instantiator/TEST-TestInstantiator-20230111221455.xml (deflated 63%) 2023-01-11T23:00:57.3325561Z adding: test/test-reports/python-unittest/distributed.test_launcher/TEST-TestDistributedLaunch-20230111221459.xml (deflated 43%) 2023-01-11T23:00:57.3326474Z adding: test/test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestGetSubmoduleToStates-20230111221503.xml (deflated 44%) 2023-01-11T23:00:57.3327674Z adding: test/test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestUtils-20230111221503.xml (deflated 69%) 2023-01-11T23:00:57.3329122Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_fx/TEST-TestSymbolicTracing-20230111221507.xml (deflated 43%) 2023-01-11T23:00:57.3330531Z adding: test/test-reports/python-unittest/distributed._tensor.test_pointwise_ops/TEST-DistElementwiseOpsTest-20230111221511.xml (deflated 71%) 2023-01-11T23:00:57.3331900Z adding: test/test-reports/python-unittest/distributed.checkpoint.test_fsdp_optim_state/TEST-FsdpOptimStateCheckpoint-20230111221515.xml (deflated 42%) 2023-01-11T23:00:57.3333045Z adding: test/test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallel-20230111221523.xml (deflated 83%) 2023-01-11T23:00:57.3334211Z adding: test/test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallelDeviceTypeCUDA-20230111221523.xml (deflated 90%) 2023-01-11T23:00:57.3335381Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal/TEST-TestTraversal-20230111221531.xml (deflated 41%) 2023-01-11T23:00:57.3336365Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_uneven/TEST-TestUnevenParamShard-20230111221539.xml (deflated 41%) 2023-01-11T23:00:57.3337425Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding_bag/TEST-TestShardedEmbeddingBag-20230111221547.xml (deflated 60%) 2023-01-11T23:00:57.3338595Z adding: test/test-reports/python-unittest/distributed._shard.sharded_optim.test_sharded_optim/TEST-TestShardedOptimizer-20230111221556.xml (deflated 59%) 2023-01-11T23:00:57.3339899Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_init/TEST-TestShardedTensorNNInit-20230111221604.xml (deflated 68%) 2023-01-11T23:00:57.3341206Z adding: test/test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestCustomShardingSpec-20230111221615.xml (deflated 66%) 2023-01-11T23:00:57.3342591Z adding: test/test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestShardingSpec-20230111221615.xml (deflated 79%) 2023-01-11T23:00:57.3343521Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_input/TEST-TestInput-20230111221627.xml (deflated 57%) 2023-01-11T23:00:57.3344721Z adding: test/test-reports/python-unittest/distributed.tensor.parallel.test_2d_parallel/TEST-Test2dParallelIntegration-20230111221639.xml (deflated 77%) 2023-01-11T23:00:57.3345874Z adding: test/test-reports/python-unittest/distributed._tensor.test_math_ops/TEST-DistMathOpsTest-20230111221654.xml (deflated 60%) 2023-01-11T23:00:57.3347031Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_tensor_ops/TEST-TestTensorOps-20230111221710.xml (deflated 76%) 2023-01-11T23:00:57.3348018Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_apply/TEST-TestApply-20230111221726.xml (deflated 61%) 2023-01-11T23:00:57.3349059Z adding: test/test-reports/python-unittest/distributed._tensor.test_tp_sharding_ops/TEST-TPShardingOpsTest-20230111221742.xml (deflated 78%) 2023-01-11T23:00:57.3350247Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_hybrid_shard/TEST-TestFSDPHybridShard-20230111221801.xml (deflated 61%) 2023-01-11T23:00:57.3351386Z adding: test/test-reports/python-unittest/distributed.test_dynamo_distributed/TEST-TestDistributed-20230111221823.xml (deflated 80%) 2023-01-11T23:00:57.3352617Z adding: test/test-reports/python-unittest/distributed.test_dynamo_distributed/TEST-TestDistributedMultiProc-20230111221823.xml (deflated 75%) 2023-01-11T23:00:57.3353930Z adding: test/test-reports/python-unittest/distributed.test_dynamo_distributed/TEST-TestFakeDistributedSingleProc-20230111221823.xml (deflated 59%) 2023-01-11T23:00:57.3355178Z adding: test/test-reports/python-unittest/distributed._tensor.test_redistribute/TEST-RedistributeTest-20230111221846.xml (deflated 75%) 2023-01-11T23:00:57.3356574Z adding: test/test-reports/python-unittest/distributed._tensor.test_redistribute/TEST-MultiDimRedistributeTest-20230111221846.xml (deflated 42%) 2023-01-11T23:00:57.3358103Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20230111221921.xml (deflated 43%) 2023-01-11T23:00:57.3359382Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20230111221925.xml (deflated 42%) 2023-01-11T23:00:57.3360226Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20230111221935.xml (deflated 42%) 2023-01-11T23:00:57.3361315Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20230111221944.xml (deflated 42%) 2023-01-11T23:00:57.3362677Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20230111221954.xml (deflated 42%) 2023-01-11T23:00:57.3363830Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20230111222004.xml (deflated 42%) 2023-01-11T23:00:57.3364976Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_meta/TEST-TestFSDPWithMetaDevice-20230111222012.xml (deflated 85%) 2023-01-11T23:00:57.3366192Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_exec_order/TEST-TestFSDPExecOrder-20230111222051.xml (deflated 83%) 2023-01-11T23:00:57.3367385Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardGradScaler-20230111222131.xml (deflated 63%) 2023-01-11T23:00:57.3368278Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardedGradScalerParityWithDDP-20230111222131.xml (deflated 83%) 2023-01-11T23:00:57.3369384Z adding: test/test-reports/python-unittest/distributed._tensor.test_dtensor/TEST-DTensorTest-20230111222211.xml (deflated 75%) 2023-01-11T23:00:57.3370781Z adding: test/test-reports/python-unittest/distributed._tensor.test_dtensor/TEST-DTensorMeshTest-20230111222211.xml (deflated 75%) 2023-01-11T23:00:57.3371959Z adding: test/test-reports/python-unittest/distributed.algorithms.test_join/TEST-TestJoin-20230111222255.xml (deflated 80%) 2023-01-11T23:00:57.3373202Z adding: test/test-reports/python-unittest/distributed._composable.test_fully_shard/TEST-TestFSDPInitialization-20230111222343.xml (deflated 72%) 2023-01-11T23:00:57.3374494Z adding: test/test-reports/python-unittest/distributed._composable.test_fully_shard/TEST-TestFSDPModelCheckpointing-20230111222343.xml (deflated 67%) 2023-01-11T23:00:57.3375752Z adding: test/test-reports/python-unittest/distributed._composable.test_fully_shard/TEST-TestFSDPRuntime-20230111222343.xml (deflated 51%) 2023-01-11T23:00:57.3376972Z adding: test/test-reports/python-unittest/distributed._composable.test_fully_shard/TEST-TestMixedPrecision-20230111222343.xml (deflated 39%) 2023-01-11T23:00:57.3378268Z adding: test/test-reports/python-unittest/distributed._composable.test_fully_shard/TEST-TestFSDPOptimStateDict-20230111222343.xml (deflated 60%) 2023-01-11T23:00:57.3379527Z adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111222440.xml (deflated 41%) 2023-01-11T23:00:57.3380871Z adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111222447.xml (deflated 41%) 2023-01-11T23:00:57.3382149Z adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111222453.xml (deflated 41%) 2023-01-11T23:00:57.3383452Z adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111222459.xml (deflated 41%) 2023-01-11T23:00:57.3384818Z adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111222506.xml (deflated 40%) 2023-01-11T23:00:57.3386348Z adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111222512.xml (deflated 41%) 2023-01-11T23:00:57.3387789Z adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111222519.xml (deflated 41%) 2023-01-11T23:00:57.3389105Z adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111222525.xml (deflated 41%) 2023-01-11T23:00:57.3390357Z adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111222531.xml (deflated 40%) 2023-01-11T23:00:57.3391668Z adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20230111222538.xml (deflated 40%) 2023-01-11T23:00:57.3392887Z adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20230111222544.xml (deflated 39%) 2023-01-11T23:00:57.3394242Z adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20230111222553.xml (deflated 39%) 2023-01-11T23:00:57.3395609Z adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20230111222602.xml (deflated 40%) 2023-01-11T23:00:57.3396984Z adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20230111222611.xml (deflated 40%) 2023-01-11T23:00:57.3398302Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint/TEST-TestFSDPCheckpoint-20230111222621.xml (deflated 90%) 2023-01-11T23:00:57.3399620Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint/TEST-TestFSDPCheckpointSubmodule-20230111222621.xml (deflated 42%) 2023-01-11T23:00:57.3400898Z adding: test/test-reports/python-unittest/distributed.test_c10d_pypg/TEST-TestDDPWithWorkSubclass-20230111222741.xml (deflated 84%) 2023-01-11T23:00:57.3402159Z adding: test/test-reports/python-unittest/distributed.test_c10d_pypg/TEST-TestDDPWithWorkWrapper-20230111222741.xml (deflated 84%) 2023-01-11T23:00:57.3403666Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsFQNs-20230111223006.xml (deflated 54%) 2023-01-11T23:00:57.3405214Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsMultipleParamGroups-20230111223006.xml (deflated 83%) 2023-01-11T23:00:57.3406629Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsNoSync-20230111223006.xml (deflated 44%) 2023-01-11T23:00:57.3408103Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsParamAccess-20230111223006.xml (deflated 45%) 2023-01-11T23:00:57.3409528Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsUnshardReshard-20230111223006.xml (deflated 76%) 2023-01-11T23:00:57.3411135Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsWriteback-20230111223006.xml (deflated 63%) 2023-01-11T23:00:57.3412626Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPDifferentSubmodulePrecision-20230111223316.xml (deflated 77%) 2023-01-11T23:00:57.3414186Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionIgnoredModules-20230111223316.xml (deflated 44%) 2023-01-11T23:00:57.3415532Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionSharded-20230111223316.xml (deflated 91%) 2023-01-11T23:00:57.3417075Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionUnsharded-20230111223316.xml (deflated 64%) 2023-01-11T23:00:57.3418696Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDdpComparisonTest-20230111223923.xml (deflated 41%) 2023-01-11T23:00:57.3420263Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20230111223930.xml (deflated 41%) 2023-01-11T23:00:57.3421706Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20230111223940.xml (deflated 41%) 2023-01-11T23:00:57.3423175Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20230111223949.xml (deflated 41%) 2023-01-11T23:00:57.3424581Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20230111223959.xml (deflated 41%) 2023-01-11T23:00:57.3425972Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20230111224008.xml (deflated 41%) 2023-01-11T23:00:57.3427373Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20230111224017.xml (deflated 41%) 2023-01-11T23:00:57.3428744Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20230111224024.xml (deflated 41%) 2023-01-11T23:00:57.3430091Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRpcTest-20230111224033.xml (deflated 40%) 2023-01-11T23:00:57.3431413Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111224045.xml (deflated 40%) 2023-01-11T23:00:57.3432751Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111224052.xml (deflated 40%) 2023-01-11T23:00:57.3434164Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111224059.xml (deflated 40%) 2023-01-11T23:00:57.3435586Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111224106.xml (deflated 40%) 2023-01-11T23:00:57.3436841Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111224113.xml (deflated 40%) 2023-01-11T23:00:57.3438210Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111224120.xml (deflated 41%) 2023-01-11T23:00:57.3439492Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111224127.xml (deflated 40%) 2023-01-11T23:00:57.3440885Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111224134.xml (deflated 40%) 2023-01-11T23:00:57.3442388Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224142.xml (deflated 42%) 2023-01-11T23:00:57.3443938Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224156.xml (deflated 42%) 2023-01-11T23:00:57.3445421Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224211.xml (deflated 42%) 2023-01-11T23:00:57.3446913Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224225.xml (deflated 44%) 2023-01-11T23:00:57.3448569Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224238.xml (deflated 44%) 2023-01-11T23:00:57.3450081Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224250.xml (deflated 43%) 2023-01-11T23:00:57.3451850Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224303.xml (deflated 43%) 2023-01-11T23:00:57.3453469Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224316.xml (deflated 44%) 2023-01-11T23:00:57.3454995Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224329.xml (deflated 43%) 2023-01-11T23:00:57.3456512Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224341.xml (deflated 43%) 2023-01-11T23:00:57.3458029Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224348.xml (deflated 43%) 2023-01-11T23:00:57.3459619Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224356.xml (deflated 43%) 2023-01-11T23:00:57.3461114Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224403.xml (deflated 43%) 2023-01-11T23:00:57.3462646Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224410.xml (deflated 43%) 2023-01-11T23:00:57.3464214Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224419.xml (deflated 44%) 2023-01-11T23:00:57.3465646Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224428.xml (deflated 42%) 2023-01-11T23:00:57.3467173Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224442.xml (deflated 43%) 2023-01-11T23:00:57.3468665Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224446.xml (deflated 43%) 2023-01-11T23:00:57.3470158Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224509.xml (deflated 43%) 2023-01-11T23:00:57.3471664Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224526.xml (deflated 43%) 2023-01-11T23:00:57.3473201Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224540.xml (deflated 43%) 2023-01-11T23:00:57.3474645Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224548.xml (deflated 42%) 2023-01-11T23:00:57.3476198Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224558.xml (deflated 42%) 2023-01-11T23:00:57.3477644Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224609.xml (deflated 42%) 2023-01-11T23:00:57.3479160Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224620.xml (deflated 43%) 2023-01-11T23:00:57.3480809Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224624.xml (deflated 43%) 2023-01-11T23:00:57.3482316Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224637.xml (deflated 42%) 2023-01-11T23:00:57.3484010Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224650.xml (deflated 43%) 2023-01-11T23:00:57.3485535Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224703.xml (deflated 42%) 2023-01-11T23:00:57.3487021Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224717.xml (deflated 42%) 2023-01-11T23:00:57.3488430Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224730.xml (deflated 43%) 2023-01-11T23:00:57.3489937Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224743.xml (deflated 42%) 2023-01-11T23:00:57.3491388Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224756.xml (deflated 42%) 2023-01-11T23:00:57.3492893Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224809.xml (deflated 43%) 2023-01-11T23:00:57.3494416Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224822.xml (deflated 43%) 2023-01-11T23:00:57.3495798Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224835.xml (deflated 42%) 2023-01-11T23:00:57.3497330Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224849.xml (deflated 42%) 2023-01-11T23:00:57.3498798Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224902.xml (deflated 42%) 2023-01-11T23:00:57.3500323Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224915.xml (deflated 42%) 2023-01-11T23:00:57.3501843Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224928.xml (deflated 42%) 2023-01-11T23:00:57.3503316Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224941.xml (deflated 42%) 2023-01-11T23:00:57.3504823Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224954.xml (deflated 42%) 2023-01-11T23:00:57.3506344Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225005.xml (deflated 43%) 2023-01-11T23:00:57.3507875Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225018.xml (deflated 42%) 2023-01-11T23:00:57.3509368Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225029.xml (deflated 42%) 2023-01-11T23:00:57.3510937Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225040.xml (deflated 43%) 2023-01-11T23:00:57.3512544Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225053.xml (deflated 43%) 2023-01-11T23:00:57.3514064Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225106.xml (deflated 43%) 2023-01-11T23:00:57.3515592Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225113.xml (deflated 43%) 2023-01-11T23:00:57.3517214Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225121.xml (deflated 43%) 2023-01-11T23:00:57.3518875Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225128.xml (deflated 42%) 2023-01-11T23:00:57.3520303Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225135.xml (deflated 42%) 2023-01-11T23:00:57.3521758Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225145.xml (deflated 42%) 2023-01-11T23:00:57.3523127Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225154.xml (deflated 42%) 2023-01-11T23:00:57.3524637Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225204.xml (deflated 42%) 2023-01-11T23:00:57.3526202Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225213.xml (deflated 42%) 2023-01-11T23:00:57.3527623Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225223.xml (deflated 42%) 2023-01-11T23:00:57.3529185Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225232.xml (deflated 42%) 2023-01-11T23:00:57.3530870Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225242.xml (deflated 43%) 2023-01-11T23:00:57.3532385Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225255.xml (deflated 42%) 2023-01-11T23:00:57.3533868Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225308.xml (deflated 43%) 2023-01-11T23:00:57.3535375Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225315.xml (deflated 43%) 2023-01-11T23:00:57.3536838Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225328.xml (deflated 43%) 2023-01-11T23:00:57.3538317Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225335.xml (deflated 43%) 2023-01-11T23:00:57.3539817Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225342.xml (deflated 42%) 2023-01-11T23:00:57.3541315Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225350.xml (deflated 43%) 2023-01-11T23:00:57.3542748Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225400.xml (deflated 42%) 2023-01-11T23:00:57.3544264Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225408.xml (deflated 42%) 2023-01-11T23:00:57.3545611Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225415.xml (deflated 42%) 2023-01-11T23:00:57.3547053Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225429.xml (deflated 42%) 2023-01-11T23:00:57.3548785Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225444.xml (deflated 41%) 2023-01-11T23:00:57.3550219Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225459.xml (deflated 42%) 2023-01-11T23:00:57.3551645Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225512.xml (deflated 42%) 2023-01-11T23:00:57.3553120Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225533.xml (deflated 42%) 2023-01-11T23:00:57.3554617Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225556.xml (deflated 42%) 2023-01-11T23:00:57.3556113Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225600.xml (deflated 42%) 2023-01-11T23:00:57.3557687Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225623.xml (deflated 42%) 2023-01-11T23:00:57.3559206Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225644.xml (deflated 42%) 2023-01-11T23:00:57.3560711Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225703.xml (deflated 42%) 2023-01-11T23:00:57.3562198Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225722.xml (deflated 42%) 2023-01-11T23:00:57.3563712Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225742.xml (deflated 42%) 2023-01-11T23:00:57.3565272Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225801.xml (deflated 42%) 2023-01-11T23:00:57.3566794Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225821.xml (deflated 42%) 2023-01-11T23:00:57.3568342Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225844.xml (deflated 42%) 2023-01-11T23:00:57.3569837Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225905.xml (deflated 42%) 2023-01-11T23:00:57.3571458Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225928.xml (deflated 42%) 2023-01-11T23:00:57.3573046Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225938.xml (deflated 43%) 2023-01-11T23:00:57.3574606Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20230111225951.xml (deflated 44%) 2023-01-11T23:00:57.3576136Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20230111225958.xml (deflated 44%) 2023-01-11T23:00:57.3577710Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20230111230005.xml (deflated 43%) 2023-01-11T23:00:57.3579150Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211344.xml (deflated 41%) 2023-01-11T23:00:57.3580770Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211350.xml (deflated 42%) 2023-01-11T23:00:57.3582268Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211356.xml (deflated 44%) 2023-01-11T23:00:57.3583674Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211359.xml (deflated 44%) 2023-01-11T23:00:57.3585108Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211401.xml (deflated 45%) 2023-01-11T23:00:57.3586530Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211403.xml (deflated 45%) 2023-01-11T23:00:57.3587911Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211407.xml (deflated 42%) 2023-01-11T23:00:57.3589201Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211413.xml (deflated 42%) 2023-01-11T23:00:57.3590603Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211419.xml (deflated 44%) 2023-01-11T23:00:57.3591994Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211421.xml (deflated 44%) 2023-01-11T23:00:57.3593383Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211423.xml (deflated 44%) 2023-01-11T23:00:57.3594797Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211425.xml (deflated 45%) 2023-01-11T23:00:57.3596179Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211439.xml (deflated 41%) 2023-01-11T23:00:57.3597462Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211448.xml (deflated 42%) 2023-01-11T23:00:57.3598678Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211451.xml (deflated 42%) 2023-01-11T23:00:57.3599950Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211457.xml (deflated 42%) 2023-01-11T23:00:57.3601236Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211502.xml (deflated 42%) 2023-01-11T23:00:57.3602462Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211508.xml (deflated 41%) 2023-01-11T23:00:57.3603805Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211515.xml (deflated 41%) 2023-01-11T23:00:57.3605165Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211524.xml (deflated 40%) 2023-01-11T23:00:57.3606396Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211533.xml (deflated 41%) 2023-01-11T23:00:57.3607680Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211541.xml (deflated 39%) 2023-01-11T23:00:57.3608953Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211550.xml (deflated 40%) 2023-01-11T23:00:57.3610180Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211559.xml (deflated 40%) 2023-01-11T23:00:57.3611468Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211608.xml (deflated 40%) 2023-01-11T23:00:57.3612947Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211616.xml (deflated 42%) 2023-01-11T23:00:57.3614279Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211620.xml (deflated 42%) 2023-01-11T23:00:57.3615447Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211627.xml (deflated 42%) 2023-01-11T23:00:57.3616652Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211631.xml (deflated 42%) 2023-01-11T23:00:57.3617916Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211639.xml (deflated 42%) 2023-01-11T23:00:57.3619185Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211642.xml (deflated 46%) 2023-01-11T23:00:57.3620423Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211649.xml (deflated 47%) 2023-01-11T23:00:57.3621677Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211655.xml (deflated 48%) 2023-01-11T23:00:57.3622862Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211702.xml (deflated 46%) 2023-01-11T23:00:57.3624171Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211709.xml (deflated 40%) 2023-01-11T23:00:57.3625349Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211716.xml (deflated 41%) 2023-01-11T23:00:57.3626608Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211723.xml (deflated 41%) 2023-01-11T23:00:57.3627793Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211729.xml (deflated 42%) 2023-01-11T23:00:57.3629115Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211736.xml (deflated 41%) 2023-01-11T23:00:57.3630255Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211743.xml (deflated 40%) 2023-01-11T23:00:57.3631398Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211750.xml (deflated 41%) 2023-01-11T23:00:57.3632543Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211756.xml (deflated 41%) 2023-01-11T23:00:57.3633815Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211759.xml (deflated 41%) 2023-01-11T23:00:57.3635013Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211801.xml (deflated 41%) 2023-01-11T23:00:57.3636164Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211808.xml (deflated 42%) 2023-01-11T23:00:57.3637422Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211815.xml (deflated 42%) 2023-01-11T23:00:57.3638634Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211817.xml (deflated 42%) 2023-01-11T23:00:57.3639874Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211820.xml (deflated 42%) 2023-01-11T23:00:57.3641061Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211822.xml (deflated 42%) 2023-01-11T23:00:57.3642332Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211824.xml (deflated 41%) 2023-01-11T23:00:57.3643645Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211831.xml (deflated 40%) 2023-01-11T23:00:57.3644858Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211838.xml (deflated 43%) 2023-01-11T23:00:57.3646167Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211840.xml (deflated 40%) 2023-01-11T23:00:57.3647389Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211847.xml (deflated 40%) 2023-01-11T23:00:57.3648663Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211854.xml (deflated 41%) 2023-01-11T23:00:57.3649832Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211901.xml (deflated 40%) 2023-01-11T23:00:57.3651099Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211907.xml (deflated 41%) 2023-01-11T23:00:57.3652189Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211914.xml (deflated 42%) 2023-01-11T23:00:57.3653427Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211921.xml (deflated 42%) 2023-01-11T23:00:57.3654680Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211928.xml (deflated 42%) 2023-01-11T23:00:57.3655845Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211934.xml (deflated 40%) 2023-01-11T23:00:57.3657009Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211941.xml (deflated 41%) 2023-01-11T23:00:57.3658182Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211948.xml (deflated 41%) 2023-01-11T23:00:57.3659407Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211955.xml (deflated 41%) 2023-01-11T23:00:57.3660633Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212002.xml (deflated 41%) 2023-01-11T23:00:57.3661748Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212008.xml (deflated 41%) 2023-01-11T23:00:57.3662934Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212015.xml (deflated 41%) 2023-01-11T23:00:57.3664204Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212022.xml (deflated 41%) 2023-01-11T23:00:57.3665371Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212029.xml (deflated 41%) 2023-01-11T23:00:57.3666632Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212035.xml (deflated 40%) 2023-01-11T23:00:57.3667812Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212042.xml (deflated 42%) 2023-01-11T23:00:57.3669119Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212049.xml (deflated 42%) 2023-01-11T23:00:57.3670265Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212056.xml (deflated 42%) 2023-01-11T23:00:57.3671345Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212103.xml (deflated 42%) 2023-01-11T23:00:57.3672883Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212109.xml (deflated 41%) 2023-01-11T23:00:57.3674256Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212116.xml (deflated 41%) 2023-01-11T23:00:57.3675546Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212123.xml (deflated 41%) 2023-01-11T23:00:57.3676703Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212131.xml (deflated 41%) 2023-01-11T23:00:57.3677974Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212140.xml (deflated 41%) 2023-01-11T23:00:57.3679150Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212147.xml (deflated 41%) 2023-01-11T23:00:57.3680359Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212155.xml (deflated 41%) 2023-01-11T23:00:57.3681623Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212202.xml (deflated 41%) 2023-01-11T23:00:57.3682829Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212209.xml (deflated 41%) 2023-01-11T23:00:57.3684074Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212216.xml (deflated 41%) 2023-01-11T23:00:57.3685246Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212226.xml (deflated 40%) 2023-01-11T23:00:57.3686404Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212236.xml (deflated 41%) 2023-01-11T23:00:57.3687564Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212246.xml (deflated 42%) 2023-01-11T23:00:57.3688731Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212248.xml (deflated 42%) 2023-01-11T23:00:57.3689947Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212250.xml (deflated 43%) 2023-01-11T23:00:57.3691302Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212253.xml (deflated 41%) 2023-01-11T23:00:57.3692542Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212255.xml (deflated 43%) 2023-01-11T23:00:57.3693832Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212258.xml (deflated 42%) 2023-01-11T23:00:57.3695008Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212300.xml (deflated 43%) 2023-01-11T23:00:57.3696270Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212302.xml (deflated 42%) 2023-01-11T23:00:57.3697447Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212305.xml (deflated 43%) 2023-01-11T23:00:57.3698754Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212307.xml (deflated 42%) 2023-01-11T23:00:57.3699987Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212310.xml (deflated 43%) 2023-01-11T23:00:57.3701174Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212312.xml (deflated 43%) 2023-01-11T23:00:57.3702461Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212314.xml (deflated 42%) 2023-01-11T23:00:57.3703993Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212317.xml (deflated 42%) 2023-01-11T23:00:57.3705315Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212319.xml (deflated 42%) 2023-01-11T23:00:57.3706507Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212322.xml (deflated 43%) 2023-01-11T23:00:57.3707808Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212324.xml (deflated 42%) 2023-01-11T23:00:57.3709062Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212326.xml (deflated 42%) 2023-01-11T23:00:57.3710241Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212329.xml (deflated 43%) 2023-01-11T23:00:57.3711515Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212331.xml (deflated 42%) 2023-01-11T23:00:57.3712566Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212334.xml (deflated 42%) 2023-01-11T23:00:57.3713702Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212336.xml (deflated 42%) 2023-01-11T23:00:57.3714861Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212338.xml (deflated 42%) 2023-01-11T23:00:57.3715826Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212341.xml (deflated 43%) 2023-01-11T23:00:57.3716572Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212343.xml (deflated 41%) 2023-01-11T23:00:57.3717341Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212353.xml (deflated 41%) 2023-01-11T23:00:57.3718091Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212400.xml (deflated 42%) 2023-01-11T23:00:57.3718845Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212402.xml (deflated 41%) 2023-01-11T23:00:57.3719583Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212410.xml (deflated 41%) 2023-01-11T23:00:57.3720335Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212419.xml (deflated 40%) 2023-01-11T23:00:57.3722007Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212426.xml (deflated 42%) 2023-01-11T23:00:57.3723256Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212433.xml (deflated 42%) 2023-01-11T23:00:57.3724014Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212440.xml (deflated 42%) 2023-01-11T23:00:57.3724777Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212447.xml (deflated 40%) 2023-01-11T23:00:57.3725531Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212454.xml (deflated 42%) 2023-01-11T23:00:57.3726289Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212457.xml (deflated 42%) 2023-01-11T23:00:57.3727030Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212504.xml (deflated 40%) 2023-01-11T23:00:57.3727866Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212510.xml (deflated 41%) 2023-01-11T23:00:57.3728681Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212517.xml (deflated 41%) 2023-01-11T23:00:57.3729435Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212519.xml (deflated 41%) 2023-01-11T23:00:57.3730181Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212522.xml (deflated 40%) 2023-01-11T23:00:57.3731137Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212524.xml (deflated 41%) 2023-01-11T23:00:57.3731891Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212527.xml (deflated 41%) 2023-01-11T23:00:57.3732649Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212529.xml (deflated 41%) 2023-01-11T23:00:57.3733396Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212531.xml (deflated 41%) 2023-01-11T23:00:57.3734151Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212534.xml (deflated 41%) 2023-01-11T23:00:57.3734902Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212536.xml (deflated 41%) 2023-01-11T23:00:57.3735652Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212543.xml (deflated 42%) 2023-01-11T23:00:57.3736393Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212547.xml (deflated 40%) 2023-01-11T23:00:57.3737147Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212554.xml (deflated 42%) 2023-01-11T23:00:57.3737911Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212601.xml (deflated 40%) 2023-01-11T23:00:57.3738665Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212609.xml (deflated 42%) 2023-01-11T23:00:57.3739402Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212613.xml (deflated 42%) 2023-01-11T23:00:57.3740157Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212617.xml (deflated 41%) 2023-01-11T23:00:57.3740912Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212621.xml (deflated 41%) 2023-01-11T23:00:57.3741667Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212631.xml (deflated 40%) 2023-01-11T23:00:57.3742416Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212641.xml (deflated 40%) 2023-01-11T23:00:57.3743173Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212651.xml (deflated 40%) 2023-01-11T23:00:57.3743928Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212700.xml (deflated 41%) 2023-01-11T23:00:57.3744682Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212708.xml (deflated 42%) 2023-01-11T23:00:57.3745420Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212712.xml (deflated 42%) 2023-01-11T23:00:57.3746173Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212716.xml (deflated 40%) 2023-01-11T23:00:57.3747089Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212725.xml (deflated 40%) 2023-01-11T23:00:57.3747852Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212733.xml (deflated 41%) 2023-01-11T23:00:57.3748589Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212742.xml (deflated 40%) 2023-01-11T23:00:57.3749341Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212750.xml (deflated 42%) 2023-01-11T23:00:57.3750093Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212755.xml (deflated 41%) 2023-01-11T23:00:57.3750844Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212801.xml (deflated 42%) 2023-01-11T23:00:57.3751583Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212806.xml (deflated 41%) 2023-01-11T23:00:57.3752341Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212814.xml (deflated 42%) 2023-01-11T23:00:57.3753093Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212819.xml (deflated 42%) 2023-01-11T23:00:57.3753845Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212823.xml (deflated 40%) 2023-01-11T23:00:57.3754602Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212832.xml (deflated 40%) 2023-01-11T23:00:57.3755338Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212841.xml (deflated 42%) 2023-01-11T23:00:57.3756091Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212846.xml (deflated 40%) 2023-01-11T23:00:57.3756885Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212855.xml (deflated 41%) 2023-01-11T23:00:57.3757641Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212904.xml (deflated 41%) 2023-01-11T23:00:57.3758377Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212913.xml (deflated 41%) 2023-01-11T23:00:57.3759129Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212922.xml (deflated 41%) 2023-01-11T23:00:57.3759878Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212931.xml (deflated 40%) 2023-01-11T23:00:57.3760634Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212943.xml (deflated 41%) 2023-01-11T23:00:57.3761375Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212959.xml (deflated 41%) 2023-01-11T23:00:57.3762126Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213008.xml (deflated 41%) 2023-01-11T23:00:57.3762875Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213017.xml (deflated 41%) 2023-01-11T23:00:57.3763625Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213026.xml (deflated 41%) 2023-01-11T23:00:57.3764364Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213035.xml (deflated 41%) 2023-01-11T23:00:57.3765114Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213044.xml (deflated 41%) 2023-01-11T23:00:57.3765987Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213053.xml (deflated 42%) 2023-01-11T23:00:57.3766751Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213058.xml (deflated 41%) 2023-01-11T23:00:57.3767486Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213107.xml (deflated 41%) 2023-01-11T23:00:57.3768235Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213115.xml (deflated 40%) 2023-01-11T23:00:57.3768986Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213122.xml (deflated 41%) 2023-01-11T23:00:57.3769741Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213131.xml (deflated 41%) 2023-01-11T23:00:57.3770648Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213139.xml (deflated 40%) 2023-01-11T23:00:57.3771419Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213158.xml (deflated 41%) 2023-01-11T23:00:57.3772170Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213207.xml (deflated 41%) 2023-01-11T23:00:57.3772922Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213216.xml (deflated 42%) 2023-01-11T23:00:57.3773663Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213224.xml (deflated 41%) 2023-01-11T23:00:57.3774416Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213233.xml (deflated 42%) 2023-01-11T23:00:57.3775175Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213237.xml (deflated 42%) 2023-01-11T23:00:57.3775929Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213241.xml (deflated 42%) 2023-01-11T23:00:57.3776668Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213251.xml (deflated 40%) 2023-01-11T23:00:57.3777420Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213259.xml (deflated 42%) 2023-01-11T23:00:57.3778173Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213303.xml (deflated 41%) 2023-01-11T23:00:57.3778926Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213312.xml (deflated 42%) 2023-01-11T23:00:57.3779837Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213316.xml (deflated 41%) 2023-01-11T23:00:57.3780595Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213326.xml (deflated 41%) 2023-01-11T23:00:57.3781342Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213334.xml (deflated 40%) 2023-01-11T23:00:57.3782096Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213342.xml (deflated 42%) 2023-01-11T23:00:57.3782831Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213346.xml (deflated 42%) 2023-01-11T23:00:57.3783583Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213350.xml (deflated 42%) 2023-01-11T23:00:57.3784425Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213354.xml (deflated 41%) 2023-01-11T23:00:57.3785252Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213403.xml (deflated 41%) 2023-01-11T23:00:57.3785988Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213412.xml (deflated 40%) 2023-01-11T23:00:57.3786740Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213419.xml (deflated 41%) 2023-01-11T23:00:57.3787491Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213426.xml (deflated 42%) 2023-01-11T23:00:57.3788246Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213430.xml (deflated 42%) 2023-01-11T23:00:57.3788990Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213434.xml (deflated 40%) 2023-01-11T23:00:57.3789746Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213441.xml (deflated 41%) 2023-01-11T23:00:57.3790498Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213447.xml (deflated 40%) 2023-01-11T23:00:57.3791253Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213454.xml (deflated 42%) 2023-01-11T23:00:57.3792010Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213457.xml (deflated 40%) 2023-01-11T23:00:57.3792748Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213504.xml (deflated 42%) 2023-01-11T23:00:57.3793504Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213510.xml (deflated 40%) 2023-01-11T23:00:57.3794264Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213517.xml (deflated 42%) 2023-01-11T23:00:57.3795020Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213521.xml (deflated 41%) 2023-01-11T23:00:57.3795759Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213528.xml (deflated 40%) 2023-01-11T23:00:57.3796515Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213535.xml (deflated 40%) 2023-01-11T23:00:57.3797263Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213542.xml (deflated 40%) 2023-01-11T23:00:57.3798017Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213549.xml (deflated 40%) 2023-01-11T23:00:57.3798767Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213556.xml (deflated 40%) 2023-01-11T23:00:57.3799524Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213604.xml (deflated 41%) 2023-01-11T23:00:57.3800275Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213611.xml (deflated 41%) 2023-01-11T23:00:57.3801031Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213618.xml (deflated 42%) 2023-01-11T23:00:57.3801768Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213625.xml (deflated 41%) 2023-01-11T23:00:57.3802521Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213632.xml (deflated 41%) 2023-01-11T23:00:57.3803389Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213657.xml (deflated 40%) 2023-01-11T23:00:57.3804153Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213723.xml (deflated 42%) 2023-01-11T23:00:57.3804888Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213729.xml (deflated 40%) 2023-01-11T23:00:57.3805641Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213738.xml (deflated 40%) 2023-01-11T23:00:57.3806390Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213745.xml (deflated 40%) 2023-01-11T23:00:57.3807141Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213752.xml (deflated 41%) 2023-01-11T23:00:57.3807880Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213759.xml (deflated 42%) 2023-01-11T23:00:57.3808636Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213801.xml (deflated 42%) 2023-01-11T23:00:57.3809388Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213804.xml (deflated 42%) 2023-01-11T23:00:57.3810137Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213806.xml (deflated 42%) 2023-01-11T23:00:57.3811052Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213809.xml (deflated 42%) 2023-01-11T23:00:57.3811809Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213811.xml (deflated 42%) 2023-01-11T23:00:57.3812565Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213813.xml (deflated 42%) 2023-01-11T23:00:57.3813326Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213816.xml (deflated 41%) 2023-01-11T23:00:57.3814063Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213818.xml (deflated 40%) 2023-01-11T23:00:57.3814811Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213825.xml (deflated 40%) 2023-01-11T23:00:57.3815565Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213832.xml (deflated 41%) 2023-01-11T23:00:57.3816318Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213834.xml (deflated 42%) 2023-01-11T23:00:57.3817062Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213837.xml (deflated 42%) 2023-01-11T23:00:57.3817821Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213841.xml (deflated 40%) 2023-01-11T23:00:57.3818576Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213849.xml (deflated 41%) 2023-01-11T23:00:57.3819330Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213859.xml (deflated 40%) 2023-01-11T23:00:57.3820070Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213908.xml (deflated 42%) 2023-01-11T23:00:57.3820824Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213913.xml (deflated 41%) 2023-01-11T23:00:57.3821577Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213917.xml (deflated 40%) 2023-01-11T23:00:57.3822478Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213924.xml (deflated 40%) 2023-01-11T23:00:57.3823229Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213930.xml (deflated 42%) 2023-01-11T23:00:57.3823983Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213935.xml (deflated 40%) 2023-01-11T23:00:57.3824734Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213941.xml (deflated 40%) 2023-01-11T23:00:57.3825487Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213948.xml (deflated 40%) 2023-01-11T23:00:57.3826228Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213955.xml (deflated 40%) 2023-01-11T23:00:57.3826996Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214002.xml (deflated 42%) 2023-01-11T23:00:57.3827751Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214009.xml (deflated 42%) 2023-01-11T23:00:57.3828504Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214016.xml (deflated 42%) 2023-01-11T23:00:57.3829242Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214022.xml (deflated 43%) 2023-01-11T23:00:57.3829998Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214029.xml (deflated 41%) 2023-01-11T23:00:57.3830754Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214036.xml (deflated 41%) 2023-01-11T23:00:57.3831513Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214043.xml (deflated 43%) 2023-01-11T23:00:57.3832249Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214045.xml (deflated 41%) 2023-01-11T23:00:57.3833008Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214052.xml (deflated 43%) 2023-01-11T23:00:57.3833758Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214054.xml (deflated 43%) 2023-01-11T23:00:57.3834505Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214057.xml (deflated 41%) 2023-01-11T23:00:57.3835242Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214104.xml (deflated 42%) 2023-01-11T23:00:57.3835999Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214106.xml (deflated 42%) 2023-01-11T23:00:57.3836759Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214109.xml (deflated 41%) 2023-01-11T23:00:57.3837511Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214115.xml (deflated 41%) 2023-01-11T23:00:57.3838264Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214122.xml (deflated 41%) 2023-01-11T23:00:57.3839007Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214129.xml (deflated 41%) 2023-01-11T23:00:57.3839764Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214136.xml (deflated 42%) 2023-01-11T23:00:57.3840569Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214138.xml (deflated 41%) 2023-01-11T23:00:57.3841376Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214141.xml (deflated 41%) 2023-01-11T23:00:57.3842114Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214147.xml (deflated 42%) 2023-01-11T23:00:57.3842870Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214154.xml (deflated 41%) 2023-01-11T23:00:57.3843620Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214201.xml (deflated 41%) 2023-01-11T23:00:57.3844371Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214208.xml (deflated 40%) 2023-01-11T23:00:57.3845112Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214214.xml (deflated 40%) 2023-01-11T23:00:57.3845872Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214221.xml (deflated 40%) 2023-01-11T23:00:57.3868422Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214228.xml (deflated 41%) 2023-01-11T23:00:57.3869338Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214235.xml (deflated 41%) 2023-01-11T23:00:57.3870132Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214237.xml (deflated 41%) 2023-01-11T23:00:57.3870895Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214240.xml (deflated 41%) 2023-01-11T23:00:57.3871675Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214242.xml (deflated 40%) 2023-01-11T23:00:57.3872460Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214249.xml (deflated 41%) 2023-01-11T23:00:57.3873236Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214256.xml (deflated 40%) 2023-01-11T23:00:57.3873998Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214302.xml (deflated 41%) 2023-01-11T23:00:57.3874769Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214309.xml (deflated 40%) 2023-01-11T23:00:57.3875530Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214316.xml (deflated 40%) 2023-01-11T23:00:57.3876296Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214324.xml (deflated 42%) 2023-01-11T23:00:57.3877051Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214333.xml (deflated 41%) 2023-01-11T23:00:57.3877809Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214340.xml (deflated 40%) 2023-01-11T23:00:57.3878575Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214348.xml (deflated 41%) 2023-01-11T23:00:57.3879346Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214357.xml (deflated 40%) 2023-01-11T23:00:57.3880101Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214410.xml (deflated 41%) 2023-01-11T23:00:57.3880866Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214426.xml (deflated 41%) 2023-01-11T23:00:57.3881848Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214436.xml (deflated 42%) 2023-01-11T23:00:57.3882631Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214438.xml (deflated 42%) 2023-01-11T23:00:57.3883381Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214445.xml (deflated 43%) 2023-01-11T23:00:57.3884148Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214449.xml (deflated 42%) 2023-01-11T23:00:57.3884915Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214456.xml (deflated 41%) 2023-01-11T23:00:57.3885674Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214503.xml (deflated 40%) 2023-01-11T23:00:57.3886423Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214512.xml (deflated 40%) 2023-01-11T23:00:57.3887201Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214520.xml (deflated 41%) 2023-01-11T23:00:57.3887967Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214529.xml (deflated 39%) 2023-01-11T23:00:57.3888729Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214537.xml (deflated 39%) 2023-01-11T23:00:57.3889471Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214547.xml (deflated 40%) 2023-01-11T23:00:57.3890241Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214555.xml (deflated 40%) 2023-01-11T23:00:57.3891233Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214604.xml (deflated 42%) 2023-01-11T23:00:57.3892004Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214608.xml (deflated 41%) 2023-01-11T23:00:57.3892753Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214615.xml (deflated 42%) 2023-01-11T23:00:57.3893522Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214619.xml (deflated 42%) 2023-01-11T23:00:57.3894283Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214627.xml (deflated 42%) 2023-01-11T23:00:57.3895050Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214629.xml (deflated 46%) 2023-01-11T23:00:57.3895797Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214636.xml (deflated 47%) 2023-01-11T23:00:57.3896569Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214643.xml (deflated 48%) 2023-01-11T23:00:57.3897327Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214650.xml (deflated 46%) 2023-01-11T23:00:57.3898090Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214657.xml (deflated 40%) 2023-01-11T23:00:57.3898835Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214704.xml (deflated 41%) 2023-01-11T23:00:57.3899600Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214710.xml (deflated 41%) 2023-01-11T23:00:57.3900363Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214717.xml (deflated 42%) 2023-01-11T23:00:57.3901286Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214724.xml (deflated 41%) 2023-01-11T23:00:57.3902064Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214731.xml (deflated 40%) 2023-01-11T23:00:57.3902809Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214738.xml (deflated 40%) 2023-01-11T23:00:57.3903573Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214744.xml (deflated 41%) 2023-01-11T23:00:57.3904335Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214747.xml (deflated 41%) 2023-01-11T23:00:57.3905097Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214749.xml (deflated 41%) 2023-01-11T23:00:57.3905857Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214756.xml (deflated 41%) 2023-01-11T23:00:57.3906622Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214803.xml (deflated 42%) 2023-01-11T23:00:57.3907383Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214805.xml (deflated 42%) 2023-01-11T23:00:57.3908140Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214808.xml (deflated 42%) 2023-01-11T23:00:57.3908887Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214810.xml (deflated 42%) 2023-01-11T23:00:57.3909649Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214812.xml (deflated 41%) 2023-01-11T23:00:57.3910422Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214819.xml (deflated 40%) 2023-01-11T23:00:57.3911185Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214826.xml (deflated 43%) 2023-01-11T23:00:57.3911931Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214828.xml (deflated 40%) 2023-01-11T23:00:57.3912692Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214835.xml (deflated 40%) 2023-01-11T23:00:57.3913456Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214842.xml (deflated 41%) 2023-01-11T23:00:57.3914218Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214849.xml (deflated 40%) 2023-01-11T23:00:57.3914966Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214856.xml (deflated 41%) 2023-01-11T23:00:57.3915733Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214902.xml (deflated 41%) 2023-01-11T23:00:57.3916492Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214909.xml (deflated 41%) 2023-01-11T23:00:57.3917257Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214916.xml (deflated 41%) 2023-01-11T23:00:57.3918004Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214922.xml (deflated 40%) 2023-01-11T23:00:57.3918768Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214929.xml (deflated 41%) 2023-01-11T23:00:57.3919576Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214936.xml (deflated 40%) 2023-01-11T23:00:57.3920476Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214943.xml (deflated 41%) 2023-01-11T23:00:57.3921225Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214949.xml (deflated 40%) 2023-01-11T23:00:57.3921989Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214956.xml (deflated 41%) 2023-01-11T23:00:57.3922756Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215003.xml (deflated 40%) 2023-01-11T23:00:57.3923513Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215010.xml (deflated 41%) 2023-01-11T23:00:57.3924262Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215017.xml (deflated 41%) 2023-01-11T23:00:57.3925026Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215024.xml (deflated 41%) 2023-01-11T23:00:57.3925789Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215031.xml (deflated 42%) 2023-01-11T23:00:57.3926545Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215037.xml (deflated 42%) 2023-01-11T23:00:57.3927290Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215044.xml (deflated 42%) 2023-01-11T23:00:57.3928049Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215051.xml (deflated 42%) 2023-01-11T23:00:57.3928811Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215058.xml (deflated 41%) 2023-01-11T23:00:57.3929581Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215104.xml (deflated 41%) 2023-01-11T23:00:57.3930477Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215111.xml (deflated 41%) 2023-01-11T23:00:57.3931262Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215120.xml (deflated 41%) 2023-01-11T23:00:57.3932026Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215128.xml (deflated 41%) 2023-01-11T23:00:57.3932789Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215135.xml (deflated 41%) 2023-01-11T23:00:57.3933531Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215144.xml (deflated 41%) 2023-01-11T23:00:57.3934308Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215150.xml (deflated 40%) 2023-01-11T23:00:57.3935071Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215157.xml (deflated 41%) 2023-01-11T23:00:57.3935831Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215204.xml (deflated 41%) 2023-01-11T23:00:57.3936577Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215214.xml (deflated 41%) 2023-01-11T23:00:57.3937343Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215224.xml (deflated 41%) 2023-01-11T23:00:57.3938101Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215234.xml (deflated 42%) 2023-01-11T23:00:57.3939013Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215236.xml (deflated 42%) 2023-01-11T23:00:57.3939770Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215239.xml (deflated 42%) 2023-01-11T23:00:57.3940533Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215241.xml (deflated 41%) 2023-01-11T23:00:57.3941295Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215243.xml (deflated 43%) 2023-01-11T23:00:57.3942056Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215246.xml (deflated 42%) 2023-01-11T23:00:57.3942804Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215248.xml (deflated 42%) 2023-01-11T23:00:57.3943573Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215250.xml (deflated 42%) 2023-01-11T23:00:57.3944343Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215253.xml (deflated 42%) 2023-01-11T23:00:57.3945109Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215255.xml (deflated 42%) 2023-01-11T23:00:57.3945854Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215258.xml (deflated 43%) 2023-01-11T23:00:57.3946607Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215300.xml (deflated 43%) 2023-01-11T23:00:57.3947368Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215302.xml (deflated 42%) 2023-01-11T23:00:57.3948132Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215305.xml (deflated 42%) 2023-01-11T23:00:57.3948901Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215307.xml (deflated 42%) 2023-01-11T23:00:57.3949646Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215310.xml (deflated 43%) 2023-01-11T23:00:57.3950408Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215312.xml (deflated 42%) 2023-01-11T23:00:57.3951169Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215314.xml (deflated 42%) 2023-01-11T23:00:57.3951926Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215317.xml (deflated 42%) 2023-01-11T23:00:57.3952674Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215319.xml (deflated 42%) 2023-01-11T23:00:57.3953442Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215321.xml (deflated 42%) 2023-01-11T23:00:57.3954202Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215324.xml (deflated 42%) 2023-01-11T23:00:57.3954958Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215326.xml (deflated 42%) 2023-01-11T23:00:57.3955705Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215329.xml (deflated 42%) 2023-01-11T23:00:57.3956469Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215331.xml (deflated 41%) 2023-01-11T23:00:57.3957273Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215341.xml (deflated 41%) 2023-01-11T23:00:57.3958143Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215347.xml (deflated 42%) 2023-01-11T23:00:57.3958898Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215350.xml (deflated 41%) 2023-01-11T23:00:57.3959658Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215357.xml (deflated 40%) 2023-01-11T23:00:57.3960417Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215406.xml (deflated 40%) 2023-01-11T23:00:57.3961180Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215414.xml (deflated 42%) 2023-01-11T23:00:57.3961926Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215421.xml (deflated 42%) 2023-01-11T23:00:57.3962698Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215428.xml (deflated 42%) 2023-01-11T23:00:57.3963460Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215434.xml (deflated 40%) 2023-01-11T23:00:57.3964223Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215442.xml (deflated 40%) 2023-01-11T23:00:57.3964970Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215450.xml (deflated 42%) 2023-01-11T23:00:57.3965728Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215457.xml (deflated 40%) 2023-01-11T23:00:57.3966487Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215504.xml (deflated 40%) 2023-01-11T23:00:57.3967260Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215511.xml (deflated 41%) 2023-01-11T23:00:57.3968005Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215513.xml (deflated 41%) 2023-01-11T23:00:57.3968761Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215515.xml (deflated 40%) 2023-01-11T23:00:57.3969515Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215518.xml (deflated 41%) 2023-01-11T23:00:57.3970269Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215520.xml (deflated 41%) 2023-01-11T23:00:57.3971182Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215523.xml (deflated 40%) 2023-01-11T23:00:57.3971938Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215525.xml (deflated 41%) 2023-01-11T23:00:57.3972702Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215528.xml (deflated 41%) 2023-01-11T23:00:57.3973452Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215530.xml (deflated 41%) 2023-01-11T23:00:57.3974186Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215537.xml (deflated 42%) 2023-01-11T23:00:57.3974937Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215541.xml (deflated 40%) 2023-01-11T23:00:57.3975698Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215548.xml (deflated 42%) 2023-01-11T23:00:57.3976537Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215555.xml (deflated 41%) 2023-01-11T23:00:57.3977351Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215603.xml (deflated 42%) 2023-01-11T23:00:57.3978115Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215607.xml (deflated 42%) 2023-01-11T23:00:57.3978877Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215611.xml (deflated 42%) 2023-01-11T23:00:57.3979642Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215615.xml (deflated 41%) 2023-01-11T23:00:57.3980374Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215626.xml (deflated 40%) 2023-01-11T23:00:57.3981143Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215635.xml (deflated 40%) 2023-01-11T23:00:57.3981911Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215645.xml (deflated 41%) 2023-01-11T23:00:57.3982672Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215654.xml (deflated 40%) 2023-01-11T23:00:57.3983413Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215703.xml (deflated 42%) 2023-01-11T23:00:57.3984178Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215707.xml (deflated 42%) 2023-01-11T23:00:57.3984935Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215711.xml (deflated 40%) 2023-01-11T23:00:57.3985696Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215719.xml (deflated 40%) 2023-01-11T23:00:57.3986448Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215727.xml (deflated 40%) 2023-01-11T23:00:57.3987206Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215736.xml (deflated 40%) 2023-01-11T23:00:57.3987969Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215745.xml (deflated 42%) 2023-01-11T23:00:57.3988732Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215749.xml (deflated 41%) 2023-01-11T23:00:57.3989474Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215756.xml (deflated 42%) 2023-01-11T23:00:57.3990237Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215800.xml (deflated 41%) 2023-01-11T23:00:57.3991009Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215809.xml (deflated 42%) 2023-01-11T23:00:57.3991770Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215813.xml (deflated 42%) 2023-01-11T23:00:57.3992512Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215817.xml (deflated 40%) 2023-01-11T23:00:57.3993275Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215826.xml (deflated 40%) 2023-01-11T23:00:57.3994011Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215836.xml (deflated 42%) 2023-01-11T23:00:57.3994743Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215840.xml (deflated 41%) 2023-01-11T23:00:57.3995533Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215849.xml (deflated 41%) 2023-01-11T23:00:57.3996326Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215858.xml (deflated 41%) 2023-01-11T23:00:57.3997062Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215907.xml (deflated 41%) 2023-01-11T23:00:57.3997794Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215916.xml (deflated 40%) 2023-01-11T23:00:57.3998534Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215925.xml (deflated 40%) 2023-01-11T23:00:57.3999272Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215934.xml (deflated 41%) 2023-01-11T23:00:57.4000021Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215943.xml (deflated 40%) 2023-01-11T23:00:57.4000789Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215952.xml (deflated 41%) 2023-01-11T23:00:57.4001547Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220001.xml (deflated 41%) 2023-01-11T23:00:57.4002286Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220010.xml (deflated 41%) 2023-01-11T23:00:57.4003023Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220019.xml (deflated 41%) 2023-01-11T23:00:57.4003765Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220028.xml (deflated 41%) 2023-01-11T23:00:57.4004504Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220037.xml (deflated 42%) 2023-01-11T23:00:57.4005243Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220041.xml (deflated 41%) 2023-01-11T23:00:57.4005994Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220050.xml (deflated 41%) 2023-01-11T23:00:57.4006728Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220059.xml (deflated 40%) 2023-01-11T23:00:57.4007471Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220106.xml (deflated 40%) 2023-01-11T23:00:57.4008215Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220115.xml (deflated 41%) 2023-01-11T23:00:57.4008973Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220123.xml (deflated 40%) 2023-01-11T23:00:57.4009738Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220141.xml (deflated 41%) 2023-01-11T23:00:57.4010747Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220150.xml (deflated 41%) 2023-01-11T23:00:57.4011512Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220159.xml (deflated 41%) 2023-01-11T23:00:57.4012270Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220208.xml (deflated 41%) 2023-01-11T23:00:57.4013031Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220217.xml (deflated 42%) 2023-01-11T23:00:57.4013792Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220221.xml (deflated 43%) 2023-01-11T23:00:57.4014688Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220225.xml (deflated 43%) 2023-01-11T23:00:57.4015464Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220235.xml (deflated 41%) 2023-01-11T23:00:57.4016228Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220243.xml (deflated 42%) 2023-01-11T23:00:57.4016983Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220247.xml (deflated 41%) 2023-01-11T23:00:57.4017727Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220256.xml (deflated 42%) 2023-01-11T23:00:57.4018486Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220300.xml (deflated 41%) 2023-01-11T23:00:57.4019256Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220310.xml (deflated 41%) 2023-01-11T23:00:57.4020016Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220318.xml (deflated 40%) 2023-01-11T23:00:57.4020763Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220326.xml (deflated 42%) 2023-01-11T23:00:57.4021526Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220330.xml (deflated 42%) 2023-01-11T23:00:57.4022286Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220335.xml (deflated 42%) 2023-01-11T23:00:57.4023052Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220339.xml (deflated 40%) 2023-01-11T23:00:57.4023801Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220348.xml (deflated 41%) 2023-01-11T23:00:57.4024571Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220356.xml (deflated 40%) 2023-01-11T23:00:57.4025329Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220403.xml (deflated 41%) 2023-01-11T23:00:57.4026092Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220409.xml (deflated 42%) 2023-01-11T23:00:57.4026831Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220414.xml (deflated 42%) 2023-01-11T23:00:57.4027592Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220418.xml (deflated 40%) 2023-01-11T23:00:57.4028356Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220425.xml (deflated 40%) 2023-01-11T23:00:57.4029123Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220431.xml (deflated 40%) 2023-01-11T23:00:57.4029868Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220438.xml (deflated 42%) 2023-01-11T23:00:57.4030631Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220441.xml (deflated 40%) 2023-01-11T23:00:57.4031394Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220448.xml (deflated 42%) 2023-01-11T23:00:57.4032157Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220455.xml (deflated 40%) 2023-01-11T23:00:57.4032955Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220501.xml (deflated 42%) 2023-01-11T23:00:57.4033777Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220506.xml (deflated 41%) 2023-01-11T23:00:57.4034541Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220512.xml (deflated 40%) 2023-01-11T23:00:57.4035306Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220519.xml (deflated 40%) 2023-01-11T23:00:57.4036047Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220526.xml (deflated 40%) 2023-01-11T23:00:57.4036808Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220533.xml (deflated 40%) 2023-01-11T23:00:57.4037571Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220539.xml (deflated 40%) 2023-01-11T23:00:57.4038336Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220548.xml (deflated 41%) 2023-01-11T23:00:57.4039098Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220555.xml (deflated 41%) 2023-01-11T23:00:57.4039847Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220602.xml (deflated 41%) 2023-01-11T23:00:57.4040602Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220608.xml (deflated 41%) 2023-01-11T23:00:57.4041358Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220615.xml (deflated 41%) 2023-01-11T23:00:57.4042118Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220640.xml (deflated 41%) 2023-01-11T23:00:57.4042871Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220705.xml (deflated 42%) 2023-01-11T23:00:57.4043616Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220712.xml (deflated 41%) 2023-01-11T23:00:57.4044366Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220721.xml (deflated 40%) 2023-01-11T23:00:57.4045114Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220728.xml (deflated 40%) 2023-01-11T23:00:57.4045860Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220735.xml (deflated 41%) 2023-01-11T23:00:57.4046613Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220742.xml (deflated 42%) 2023-01-11T23:00:57.4047377Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220744.xml (deflated 42%) 2023-01-11T23:00:57.4048124Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220747.xml (deflated 42%) 2023-01-11T23:00:57.4048862Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220749.xml (deflated 42%) 2023-01-11T23:00:57.4049614Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220751.xml (deflated 42%) 2023-01-11T23:00:57.4050525Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220754.xml (deflated 42%) 2023-01-11T23:00:57.4051295Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220756.xml (deflated 42%) 2023-01-11T23:00:57.4052101Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220759.xml (deflated 41%) 2023-01-11T23:00:57.4052935Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220801.xml (deflated 40%) 2023-01-11T23:00:57.4053689Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220808.xml (deflated 41%) 2023-01-11T23:00:57.4054453Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220815.xml (deflated 41%) 2023-01-11T23:00:57.4055193Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220817.xml (deflated 42%) 2023-01-11T23:00:57.4055950Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220819.xml (deflated 42%) 2023-01-11T23:00:57.4056713Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220824.xml (deflated 40%) 2023-01-11T23:00:57.4057519Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220832.xml (deflated 41%) 2023-01-11T23:00:57.4058265Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220842.xml (deflated 40%) 2023-01-11T23:00:57.4059024Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220851.xml (deflated 42%) 2023-01-11T23:00:57.4059780Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220856.xml (deflated 41%) 2023-01-11T23:00:57.4060542Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220900.xml (deflated 40%) 2023-01-11T23:00:57.4061287Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220906.xml (deflated 40%) 2023-01-11T23:00:57.4062053Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220913.xml (deflated 42%) 2023-01-11T23:00:57.4062816Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220917.xml (deflated 40%) 2023-01-11T23:00:57.4063575Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220924.xml (deflated 40%) 2023-01-11T23:00:57.4064322Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220931.xml (deflated 40%) 2023-01-11T23:00:57.4065084Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220938.xml (deflated 40%) 2023-01-11T23:00:57.4065838Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220945.xml (deflated 42%) 2023-01-11T23:00:57.4066605Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220951.xml (deflated 42%) 2023-01-11T23:00:57.4067351Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220958.xml (deflated 42%) 2023-01-11T23:00:57.4068115Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221005.xml (deflated 42%) 2023-01-11T23:00:57.4068868Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221012.xml (deflated 41%) 2023-01-11T23:00:57.4069627Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221019.xml (deflated 41%) 2023-01-11T23:00:57.4070368Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221025.xml (deflated 43%) 2023-01-11T23:00:57.4071231Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221028.xml (deflated 41%) 2023-01-11T23:00:57.4072001Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221035.xml (deflated 43%) 2023-01-11T23:00:57.4072756Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221037.xml (deflated 44%) 2023-01-11T23:00:57.4073501Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221039.xml (deflated 41%) 2023-01-11T23:00:57.4074258Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221046.xml (deflated 42%) 2023-01-11T23:00:57.4075013Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221049.xml (deflated 42%) 2023-01-11T23:00:57.4075782Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221051.xml (deflated 41%) 2023-01-11T23:00:57.4076528Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221058.xml (deflated 41%) 2023-01-11T23:00:57.4077286Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221105.xml (deflated 41%) 2023-01-11T23:00:57.4078044Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221112.xml (deflated 41%) 2023-01-11T23:00:57.4078807Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221118.xml (deflated 42%) 2023-01-11T23:00:57.4079550Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221121.xml (deflated 42%) 2023-01-11T23:00:57.4080316Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221123.xml (deflated 40%) 2023-01-11T23:00:57.4081080Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221130.xml (deflated 42%) 2023-01-11T23:00:57.4081844Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221137.xml (deflated 41%) 2023-01-11T23:00:57.4082582Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221144.xml (deflated 41%) 2023-01-11T23:00:57.4083338Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221150.xml (deflated 41%) 2023-01-11T23:00:57.4084094Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221157.xml (deflated 40%) 2023-01-11T23:00:57.4084855Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221204.xml (deflated 41%) 2023-01-11T23:00:57.4085615Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221211.xml (deflated 41%) 2023-01-11T23:00:57.4086356Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221217.xml (deflated 42%) 2023-01-11T23:00:57.4087112Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221220.xml (deflated 41%) 2023-01-11T23:00:57.4087878Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221222.xml (deflated 41%) 2023-01-11T23:00:57.4088636Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221225.xml (deflated 41%) 2023-01-11T23:00:57.4089431Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221231.xml (deflated 41%) 2023-01-11T23:00:57.4090248Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221238.xml (deflated 41%) 2023-01-11T23:00:57.4091171Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221245.xml (deflated 41%) 2023-01-11T23:00:57.4091934Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221252.xml (deflated 41%) 2023-01-11T23:00:57.4092681Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221259.xml (deflated 41%) 2023-01-11T23:00:57.4093440Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221307.xml (deflated 41%) 2023-01-11T23:00:57.4094198Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221316.xml (deflated 41%) 2023-01-11T23:00:57.4094967Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221322.xml (deflated 40%) 2023-01-11T23:00:57.4095712Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221331.xml (deflated 41%) 2023-01-11T23:00:57.4096471Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221339.xml (deflated 40%) 2023-01-11T23:00:57.4097231Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221353.xml (deflated 40%) 2023-01-11T23:00:57.4097879Z adding: test/test-reports/cpp-distributed/test_distributed/FileStoreTest.xml (deflated 72%) 2023-01-11T23:00:57.4098443Z adding: test/test-reports/cpp-distributed/test_distributed/HashStoreTest.xml (deflated 71%) 2023-01-11T23:00:57.4099019Z adding: test/test-reports/cpp-distributed/test_distributed/TCPStoreTest.xml (deflated 80%) 2023-01-11T23:00:57.4099622Z adding: test/test-reports/cpp-distributed/test_distributed/ProcessGroupGlooTest.xml (deflated 80%) 2023-01-11T23:00:57.4100219Z adding: test/test-reports/cpp-distributed/test_distributed/ProcessGroupNCCLTest.xml (deflated 80%) 2023-01-11T23:00:57.4100862Z adding: test/test-reports/cpp-distributed/test_distributed/ProcessGroupNCCLErrorsTest.xml (deflated 67%) 2023-01-11T23:00:57.4101429Z adding: test/test-reports/cpp-rpc/test_rpc/test_cpp_rpc.xml (deflated 78%) 2023-01-11T23:00:57.4123485Z ##[group]Run # Remove any previous test reports if they exist 2023-01-11T23:00:57.4123867Z # Remove any previous test reports if they exist 2023-01-11T23:00:57.4124162Z rm -f usage-log-*.zip 2023-01-11T23:00:57.4124524Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2023-01-11T23:00:57.4124907Z # so check to see if the file exists first 2023-01-11T23:00:57.4125204Z if [ -f 'usage_log.txt' ]; then 2023-01-11T23:00:57.4125534Z  zip "usage-log-${FILE_SUFFIX}.zip" 'usage_log.txt' 2023-01-11T23:00:57.4125822Z fi 2023-01-11T23:00:57.4138194Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-01-11T23:00:57.4138469Z env: 2023-01-11T23:00:57.4138710Z GIT_DEFAULT_BRANCH: master 2023-01-11T23:00:57.4138975Z GPU_FLAG: --gpus all 2023-01-11T23:00:57.4139299Z DOCKER_CONTAINER_ID: 76a1849f93d45a34422360460369a49f22eebafa5f0784e84d188aa13b21145b 2023-01-11T23:00:57.4139752Z FILE_SUFFIX: test-distributed-1-3-linux.8xlarge.nvidia.gpu_10589291239 2023-01-11T23:00:57.4140101Z ##[endgroup] 2023-01-11T23:00:57.5093660Z adding: usage_log.txt 2023-01-11T23:00:57.5094505Z zip warning: file size changed while zipping usage_log.txt 2023-01-11T23:00:57.5095049Z (deflated 95%) 2023-01-11T23:00:57.5141687Z ##[group]Run seemethere/upload-artifact-s3@v5 2023-01-11T23:00:57.5141973Z with: 2023-01-11T23:00:57.5142247Z s3-prefix: pytorch/pytorch/3896099317/2/artifact 2023-01-11T23:00:57.5142726Z retention-days: 14 2023-01-11T23:00:57.5142993Z if-no-files-found: warn 2023-01-11T23:00:57.5143262Z path: test-jsons-*.zip 2023-01-11T23:00:57.5143495Z name: artifact 2023-01-11T23:00:57.5143739Z s3-bucket: gha-artifacts 2023-01-11T23:00:57.5143997Z region: us-east-1 2023-01-11T23:00:57.5144204Z env: 2023-01-11T23:00:57.5144438Z GIT_DEFAULT_BRANCH: master 2023-01-11T23:00:57.5144699Z GPU_FLAG: --gpus all 2023-01-11T23:00:57.5145022Z DOCKER_CONTAINER_ID: 76a1849f93d45a34422360460369a49f22eebafa5f0784e84d188aa13b21145b 2023-01-11T23:00:57.5145347Z ##[endgroup] 2023-01-11T23:00:57.9560073Z NOTE: s3-prefix specified, ignoring name parameter 2023-01-11T23:00:57.9560960Z With the provided path, there will be 1 file uploaded 2023-01-11T23:00:57.9561343Z Uploading to s3 prefix: pytorch/pytorch/3896099317/2/artifact 2023-01-11T23:00:57.9572494Z Starting upload of test-jsons-test-distributed-1-3-linux.8xlarge.nvidia.gpu_10589291239.zip 2023-01-11T23:00:58.0946716Z Finished upload of test-jsons-test-distributed-1-3-linux.8xlarge.nvidia.gpu_10589291239.zip 2023-01-11T23:00:58.1144103Z ##[group]Run seemethere/upload-artifact-s3@v5 2023-01-11T23:00:58.1144396Z with: 2023-01-11T23:00:58.1144668Z s3-prefix: pytorch/pytorch/3896099317/2/artifact 2023-01-11T23:00:58.1144945Z retention-days: 14 2023-01-11T23:00:58.1145215Z if-no-files-found: error 2023-01-11T23:00:58.1145492Z path: test-reports-*.zip 2023-01-11T23:00:58.1145729Z name: artifact 2023-01-11T23:00:58.1145977Z s3-bucket: gha-artifacts 2023-01-11T23:00:58.1146233Z region: us-east-1 2023-01-11T23:00:58.1146444Z env: 2023-01-11T23:00:58.1146679Z GIT_DEFAULT_BRANCH: master 2023-01-11T23:00:58.1146943Z GPU_FLAG: --gpus all 2023-01-11T23:00:58.1147266Z DOCKER_CONTAINER_ID: 76a1849f93d45a34422360460369a49f22eebafa5f0784e84d188aa13b21145b 2023-01-11T23:00:58.1147594Z ##[endgroup] 2023-01-11T23:00:58.5595703Z NOTE: s3-prefix specified, ignoring name parameter 2023-01-11T23:00:58.5596406Z With the provided path, there will be 1 file uploaded 2023-01-11T23:00:58.5596783Z Uploading to s3 prefix: pytorch/pytorch/3896099317/2/artifact 2023-01-11T23:00:58.5607792Z Starting upload of test-reports-test-distributed-1-3-linux.8xlarge.nvidia.gpu_10589291239.zip 2023-01-11T23:00:58.6850337Z Finished upload of test-reports-test-distributed-1-3-linux.8xlarge.nvidia.gpu_10589291239.zip 2023-01-11T23:00:58.7029594Z ##[group]Run seemethere/upload-artifact-s3@v5 2023-01-11T23:00:58.7029881Z with: 2023-01-11T23:00:58.7030153Z s3-prefix: pytorch/pytorch/3896099317/2/artifact 2023-01-11T23:00:58.7030434Z retention-days: 14 2023-01-11T23:00:58.7030696Z if-no-files-found: ignore 2023-01-11T23:00:58.7030967Z path: usage-log-*.zip 2023-01-11T23:00:58.7031194Z name: artifact 2023-01-11T23:00:58.7031442Z s3-bucket: gha-artifacts 2023-01-11T23:00:58.7031696Z region: us-east-1 2023-01-11T23:00:58.7031902Z env: 2023-01-11T23:00:58.7032134Z GIT_DEFAULT_BRANCH: master 2023-01-11T23:00:58.7032400Z GPU_FLAG: --gpus all 2023-01-11T23:00:58.7032727Z DOCKER_CONTAINER_ID: 76a1849f93d45a34422360460369a49f22eebafa5f0784e84d188aa13b21145b 2023-01-11T23:00:58.7033053Z ##[endgroup] 2023-01-11T23:00:59.1468870Z NOTE: s3-prefix specified, ignoring name parameter 2023-01-11T23:00:59.1469624Z With the provided path, there will be 1 file uploaded 2023-01-11T23:00:59.1469986Z Uploading to s3 prefix: pytorch/pytorch/3896099317/2/artifact 2023-01-11T23:00:59.1481369Z Starting upload of usage-log-test-distributed-1-3-linux.8xlarge.nvidia.gpu_10589291239.zip 2023-01-11T23:00:59.3336473Z Finished upload of usage-log-test-distributed-1-3-linux.8xlarge.nvidia.gpu_10589291239.zip 2023-01-11T23:00:59.3510386Z ##[group]Run # shellcheck disable=SC2156 2023-01-11T23:00:59.3510737Z # shellcheck disable=SC2156 2023-01-11T23:00:59.3511130Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2023-01-11T23:00:59.3524693Z shell: /usr/bin/bash -e {0} 2023-01-11T23:00:59.3525090Z env: 2023-01-11T23:00:59.3525330Z GIT_DEFAULT_BRANCH: master 2023-01-11T23:00:59.3525579Z GPU_FLAG: --gpus all 2023-01-11T23:00:59.3525924Z DOCKER_CONTAINER_ID: 76a1849f93d45a34422360460369a49f22eebafa5f0784e84d188aa13b21145b 2023-01-11T23:00:59.3526251Z ##[endgroup] 2023-01-11T23:00:59.6792323Z ##[group]Run set -x 2023-01-11T23:00:59.6792635Z set -x 2023-01-11T23:00:59.6792915Z python3 -m pip install -r requirements.txt 2023-01-11T23:00:59.6793252Z python3 -m pip install boto3==1.19.12 2023-01-11T23:00:59.6793642Z python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test 2023-01-11T23:00:59.6806451Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-01-11T23:00:59.6806747Z env: 2023-01-11T23:00:59.6806989Z GIT_DEFAULT_BRANCH: master 2023-01-11T23:00:59.6807238Z GPU_FLAG: --gpus all 2023-01-11T23:00:59.6807585Z DOCKER_CONTAINER_ID: 76a1849f93d45a34422360460369a49f22eebafa5f0784e84d188aa13b21145b 2023-01-11T23:00:59.6807946Z AWS_DEFAULT_REGION: us-east-1 2023-01-11T23:00:59.6808194Z BRANCH: pull/91627 2023-01-11T23:00:59.6808452Z TEST_CONFIG: distributed 2023-01-11T23:00:59.6808704Z SHARD_NUMBER: 1 2023-01-11T23:00:59.6809018Z BUILD_ENVIRONMENT: linux-bionic-cuda11.6-py3.10-gcc7 2023-01-11T23:00:59.6809351Z PR_NUMBER: 91627 2023-01-11T23:00:59.6809612Z PYTORCH_RETRY_TEST_CASES: 1 2023-01-11T23:00:59.6809877Z PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1 2023-01-11T23:00:59.6810190Z SHA1: 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e 2023-01-11T23:00:59.6810731Z TAG: 2023-01-11T23:00:59.6810946Z WORKFLOW_ID: 3896099317 2023-01-11T23:00:59.6811388Z GITHUB_TOKEN: *** 2023-01-11T23:00:59.6811651Z GHA_WORKFLOW_JOB_ID: 10589291239 2023-01-11T23:00:59.6811890Z ##[endgroup] 2023-01-11T23:00:59.6841173Z + python3 -m pip install -r requirements.txt 2023-01-11T23:00:59.9769884Z Defaulting to user installation because normal site-packages is not writeable 2023-01-11T23:01:00.0634488Z Collecting astunparse 2023-01-11T23:01:00.0807466Z Downloading astunparse-1.6.3-py2.py3-none-any.whl (12 kB) 2023-01-11T23:01:00.1137040Z Collecting expecttest 2023-01-11T23:01:00.1189266Z Downloading expecttest-0.1.4-py3-none-any.whl (6.5 kB) 2023-01-11T23:01:00.1583670Z Collecting future 2023-01-11T23:01:00.1630274Z Downloading future-0.18.2.tar.gz (829 kB) 2023-01-11T23:01:02.0482146Z Collecting hypothesis 2023-01-11T23:01:02.0573128Z Downloading hypothesis-6.62.0-py3-none-any.whl (399 kB) 2023-01-11T23:01:02.7722747Z Collecting numpy 2023-01-11T23:01:02.7810595Z Downloading numpy-1.21.6-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB) 2023-01-11T23:01:03.1436636Z Requirement already satisfied: psutil in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 7)) (5.9.1) 2023-01-11T23:01:03.2868554Z Collecting pyyaml 2023-01-11T23:01:03.2926665Z Downloading PyYAML-6.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (596 kB) 2023-01-11T23:01:03.3228044Z Requirement already satisfied: requests in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 9)) (2.26.0) 2023-01-11T23:01:03.3394868Z Requirement already satisfied: setuptools in /usr/lib/python3.7/site-packages (from -r requirements.txt (line 10)) (49.1.3) 2023-01-11T23:01:03.4200555Z Collecting six 2023-01-11T23:01:03.4248083Z Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) 2023-01-11T23:01:03.4933206Z Collecting types-dataclasses 2023-01-11T23:01:03.4990947Z Downloading types_dataclasses-0.6.6-py3-none-any.whl (2.9 kB) 2023-01-11T23:01:03.5656187Z Collecting typing_extensions 2023-01-11T23:01:03.5702779Z Downloading typing_extensions-4.4.0-py3-none-any.whl (26 kB) 2023-01-11T23:01:03.6296579Z Collecting sympy 2023-01-11T23:01:03.6376702Z Downloading sympy-1.10.1-py3-none-any.whl (6.4 MB) 2023-01-11T23:01:03.8362109Z Collecting filelock 2023-01-11T23:01:03.8409493Z Downloading filelock-3.9.0-py3-none-any.whl (9.7 kB) 2023-01-11T23:01:03.9367135Z Collecting networkx 2023-01-11T23:01:03.9442982Z Downloading networkx-2.6.3-py3-none-any.whl (1.9 MB) 2023-01-11T23:01:04.0659688Z Collecting jinja2 2023-01-11T23:01:04.0722725Z Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB) 2023-01-11T23:01:04.1680650Z Collecting wheel<1.0,>=0.23.0 2023-01-11T23:01:04.1728666Z Downloading wheel-0.38.4-py3-none-any.whl (36 kB) 2023-01-11T23:01:04.2195773Z Collecting sortedcontainers<3.0.0,>=2.1.0 2023-01-11T23:01:04.2276365Z Downloading sortedcontainers-2.4.0-py2.py3-none-any.whl (29 kB) 2023-01-11T23:01:04.2675498Z Collecting exceptiongroup>=1.0.0; python_version < "3.11" 2023-01-11T23:01:04.2722250Z Downloading exceptiongroup-1.1.0-py3-none-any.whl (14 kB) 2023-01-11T23:01:04.3248641Z Collecting attrs>=19.2.0 2023-01-11T23:01:04.3295547Z Downloading attrs-22.2.0-py3-none-any.whl (60 kB) 2023-01-11T23:01:04.3760677Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (2.0.12) 2023-01-11T23:01:04.3785617Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (3.4) 2023-01-11T23:01:04.3800089Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (1.26.14) 2023-01-11T23:01:04.4014671Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (2022.12.7) 2023-01-11T23:01:04.4287808Z Collecting mpmath>=0.19 2023-01-11T23:01:04.4361241Z Downloading mpmath-1.2.1-py3-none-any.whl (532 kB) 2023-01-11T23:01:04.5890123Z Collecting MarkupSafe>=2.0 2023-01-11T23:01:04.5939836Z Downloading MarkupSafe-2.1.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB) 2023-01-11T23:01:04.6033508Z Using legacy 'setup.py install' for future, since package 'wheel' is not installed. 2023-01-11T23:01:04.7855376Z Installing collected packages: six, wheel, astunparse, expecttest, future, sortedcontainers, exceptiongroup, attrs, hypothesis, numpy, pyyaml, types-dataclasses, typing-extensions, mpmath, sympy, filelock, networkx, MarkupSafe, jinja2 2023-01-11T23:01:04.8278594Z WARNING: The script wheel is installed in '/home/ec2-user/.local/bin' which is not on PATH. 2023-01-11T23:01:04.8279463Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2023-01-11T23:01:04.8597840Z Running setup.py install for future: started 2023-01-11T23:01:05.5190133Z Running setup.py install for future: finished with status 'done' 2023-01-11T23:01:05.8298566Z WARNING: The script hypothesis is installed in '/home/ec2-user/.local/bin' which is not on PATH. 2023-01-11T23:01:05.8299329Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2023-01-11T23:01:07.8182307Z WARNING: The scripts f2py, f2py3 and f2py3.7 are installed in '/home/ec2-user/.local/bin' which is not on PATH. 2023-01-11T23:01:07.8183013Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2023-01-11T23:01:16.8349384Z WARNING: The script isympy is installed in '/home/ec2-user/.local/bin' which is not on PATH. 2023-01-11T23:01:16.8350054Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2023-01-11T23:01:18.0954952Z Successfully installed MarkupSafe-2.1.1 astunparse-1.6.3 attrs-22.2.0 exceptiongroup-1.1.0 expecttest-0.1.4 filelock-3.9.0 future-0.18.2 hypothesis-6.62.0 jinja2-3.1.2 mpmath-1.2.1 networkx-2.6.3 numpy-1.21.6 pyyaml-6.0 six-1.16.0 sortedcontainers-2.4.0 sympy-1.10.1 types-dataclasses-0.6.6 typing-extensions-4.4.0 wheel-0.38.4 2023-01-11T23:01:18.1720904Z + python3 -m pip install boto3==1.19.12 2023-01-11T23:01:18.4683467Z Defaulting to user installation because normal site-packages is not writeable 2023-01-11T23:01:19.4529804Z Collecting boto3==1.19.12 2023-01-11T23:01:19.4743049Z Downloading boto3-1.19.12-py3-none-any.whl (131 kB) 2023-01-11T23:01:19.5373684Z Collecting s3transfer<0.6.0,>=0.5.0 2023-01-11T23:01:19.5420897Z Downloading s3transfer-0.5.2-py3-none-any.whl (79 kB) 2023-01-11T23:01:19.6007666Z Collecting jmespath<1.0.0,>=0.7.1 2023-01-11T23:01:19.6056246Z Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB) 2023-01-11T23:01:20.7540572Z Collecting botocore<1.23.0,>=1.22.12 2023-01-11T23:01:20.7611206Z Downloading botocore-1.22.12-py3-none-any.whl (8.1 MB) 2023-01-11T23:01:20.9907073Z Collecting python-dateutil<3.0.0,>=2.1 2023-01-11T23:01:20.9957017Z Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) 2023-01-11T23:01:21.0137081Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.26.14) 2023-01-11T23:01:21.0356125Z Requirement already satisfied: six>=1.5 in /home/ec2-user/.local/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.16.0) 2023-01-11T23:01:21.2480083Z Installing collected packages: jmespath, python-dateutil, botocore, s3transfer, boto3 2023-01-11T23:01:22.1425654Z Successfully installed boto3-1.19.12 botocore-1.22.12 jmespath-0.10.0 python-dateutil-2.8.2 s3transfer-0.5.2 2023-01-11T23:01:22.1978670Z + python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test 2023-01-11T23:02:55.8598538Z [scribe] Scribe access token not provided, sending report via boto3... 2023-01-11T23:02:55.8599393Z ERROR ENCOUNTERED WHEN UPLOADING TO SCRIBE: {"errorMessage":"2023-01-11T23:02:22.954Z b8685fd7-6c5d-42d9-a9df-bf16d3add7d7 Task timed out after 60.00 seconds"} 2023-01-11T23:02:55.8599729Z 2023-01-11T23:02:55.8602887Z ----- Historic stats comparison result ------ 2023-01-11T23:02:55.8603150Z 2023-01-11T23:02:55.8603419Z job: linux-bionic-cuda11.6-py3.10-gcc7 2023-01-11T23:02:55.8603779Z commit: 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e 2023-01-11T23:02:55.8603987Z 2023-01-11T23:02:55.8604193Z Commit graph (base is most recent master ancestor with at least one S3 report): 2023-01-11T23:02:55.8604438Z 2023-01-11T23:02:55.8604535Z : (master) 2023-01-11T23:02:55.8604771Z | 2023-01-11T23:02:55.8606485Z | * 8419ddda87 (HEAD) total time 3424.75s 2023-01-11T23:02:55.8606854Z | | 2023-01-11T23:02:55.8607080Z | : (2 commits) 2023-01-11T23:02:55.8607313Z |/ 2023-01-11T23:02:55.8607873Z * db2a237763 (base) 18 reports, total time 5298.55s ± 3363.99s 2023-01-11T23:02:55.8608329Z * 2b0abd4ce3 18 reports, total time 5375.52s ± 3469.84s 2023-01-11T23:02:55.8615972Z * f7939b21e1 48 reports, total time 4156.05s ± 3748.80s 2023-01-11T23:02:55.8616442Z * cb3204823e 18 reports, total time 5386.78s ± 3494.12s 2023-01-11T23:02:55.8616869Z * 6e236553f5 18 reports, total time 5359.08s ± 3471.03s 2023-01-11T23:02:55.8617388Z * cce577b391 18 reports, total time 5383.89s ± 3498.12s 2023-01-11T23:02:55.8617793Z * fae821c2f1 18 reports, total time 5181.34s ± 3399.30s 2023-01-11T23:02:55.8618224Z * 0c3659586d 18 reports, total time 5181.30s ± 3409.47s 2023-01-11T23:02:55.8618639Z * 122245985a 18 reports, total time 5175.63s ± 3372.26s 2023-01-11T23:02:55.8619030Z * b797a24259 18 reports, total time 5151.22s ± 3376.21s 2023-01-11T23:02:55.8619309Z | 2023-01-11T23:02:55.8619530Z : 2023-01-11T23:02:55.8619668Z 2023-01-11T23:02:55.8619836Z Removed (across 1477 suites) 0 tests, totaling 0.00s 2023-01-11T23:02:55.8620166Z Modified (across 0 suites) 0 tests, totaling 0.00s 2023-01-11T23:02:55.8620514Z Added (across 78 suites) 822 tests, totaling +4492.15s 2023-01-11T23:02:55.9243610Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2023-01-11T23:02:55.9244112Z with: 2023-01-11T23:02:55.9244305Z env: 2023-01-11T23:02:55.9244541Z GIT_DEFAULT_BRANCH: master 2023-01-11T23:02:55.9244804Z GPU_FLAG: --gpus all 2023-01-11T23:02:55.9245128Z DOCKER_CONTAINER_ID: 76a1849f93d45a34422360460369a49f22eebafa5f0784e84d188aa13b21145b 2023-01-11T23:02:55.9245456Z ##[endgroup] 2023-01-11T23:02:55.9265058Z ##[group]Run set -eou pipefail 2023-01-11T23:02:55.9265342Z set -eou pipefail 2023-01-11T23:02:55.9265586Z  2023-01-11T23:02:55.9265902Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2023-01-11T23:02:55.9266218Z for _ in $(seq 1440); do 2023-01-11T23:02:55.9266510Z  # Break if no ssh session exists anymore 2023-01-11T23:02:55.9266802Z  if [ "$(who)" = "" ]; then 2023-01-11T23:02:55.9267050Z  break 2023-01-11T23:02:55.9267298Z  fi 2023-01-11T23:02:55.9267536Z  echo "." 2023-01-11T23:02:55.9267756Z  sleep 5 2023-01-11T23:02:55.9267996Z done 2023-01-11T23:02:55.9281479Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-01-11T23:02:55.9281757Z env: 2023-01-11T23:02:55.9281998Z GIT_DEFAULT_BRANCH: master 2023-01-11T23:02:55.9282264Z GPU_FLAG: --gpus all 2023-01-11T23:02:55.9282589Z DOCKER_CONTAINER_ID: 76a1849f93d45a34422360460369a49f22eebafa5f0784e84d188aa13b21145b 2023-01-11T23:02:55.9282919Z ##[endgroup] 2023-01-11T23:02:55.9312263Z Holding runner for 2 hours until all ssh sessions have logged out 2023-01-11T23:02:55.9385094Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2023-01-11T23:02:55.9385500Z # ignore expansion of "docker ps -q" since it could be empty 2023-01-11T23:02:55.9385837Z # shellcheck disable=SC2046 2023-01-11T23:02:55.9386139Z docker stop $(docker ps -q) || true 2023-01-11T23:02:55.9386454Z # Prune all of the docker images 2023-01-11T23:02:55.9386729Z docker system prune -af 2023-01-11T23:02:55.9399135Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-01-11T23:02:55.9399434Z env: 2023-01-11T23:02:55.9399679Z GIT_DEFAULT_BRANCH: master 2023-01-11T23:02:55.9399933Z GPU_FLAG: --gpus all 2023-01-11T23:02:55.9400279Z DOCKER_CONTAINER_ID: 76a1849f93d45a34422360460369a49f22eebafa5f0784e84d188aa13b21145b 2023-01-11T23:02:55.9400609Z ##[endgroup] 2023-01-11T23:02:56.5399176Z 76a1849f93d4 2023-01-11T23:02:57.6389187Z Deleted Containers: 2023-01-11T23:02:57.6389565Z 76a1849f93d45a34422360460369a49f22eebafa5f0784e84d188aa13b21145b 2023-01-11T23:02:57.6389779Z 2023-01-11T23:03:02.6299231Z Deleted Images: 2023-01-11T23:03:02.6300080Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd 2023-01-11T23:03:02.6301053Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7@sha256:866df6c1171dbe014496717cf2080d6cc72ca611a4e8146525c9ef09640c8ba4 2023-01-11T23:03:02.6301657Z deleted: sha256:09e297797cd8c095524ba49e041c45e57bf05ba16719f01e7240e8549da5beba 2023-01-11T23:03:02.6302066Z deleted: sha256:6d5f0082fbf8c3b01e49961283f44105a5bb12616f0073762021db97f20a16a5 2023-01-11T23:03:02.6302494Z deleted: sha256:19c574c96e47e3d16cc51cc088fed6cef16eaa2174e667ba0b395ac0e3b989bb 2023-01-11T23:03:02.6302949Z deleted: sha256:4fa7af758c23581dbcc2bd92defdc5fea97c8671fa67bbc30888ffbbf96c49a6 2023-01-11T23:03:02.6303396Z deleted: sha256:e68331e1b0f863bbdbd445ed8475d59d2234c9659264a4e49b7b096311445aee 2023-01-11T23:03:02.6303808Z deleted: sha256:69d886418998cd8758c555ed219fca3a457539e9d6f62c41c9664a80c82c4036 2023-01-11T23:03:02.6304211Z deleted: sha256:2368b1fdf0235d155eaa47e75ff379cff06cc82d63e03886a57363b8092d3c83 2023-01-11T23:03:02.6304648Z deleted: sha256:651c1e1b625aeaf8fa65e96ef11487e650a7a6d00ba8ec9fdfe7e89e762dc5c4 2023-01-11T23:03:02.6305306Z deleted: sha256:7b73f298df08c4b3aa849b55bb5e73ded619cb5b786fddcc21ece3c0b3887038 2023-01-11T23:03:02.6305841Z deleted: sha256:716f28b4b433958e5d1b9839a20dd22f9986f9c1b42fb95552f13d6bfd291efe 2023-01-11T23:03:02.6306262Z deleted: sha256:e4495993276176c504228e66b2fd6f348523c5af66f9292b2f7ea12acefcd606 2023-01-11T23:03:02.6306659Z deleted: sha256:d6f5fbd8782783697c73f4bcbce91a05e126575f72f0f58e1e9465aa57640a92 2023-01-11T23:03:02.6307067Z deleted: sha256:0b7725c897ee2681e3b1ff00aea6c14805c8050758cfa1010f561c9713934014 2023-01-11T23:03:02.6307476Z deleted: sha256:fdd864c6750bdd24f8cfb131673c6f04087e3cbecac2c1a9b3c30fefbd6d3070 2023-01-11T23:03:02.6307939Z deleted: sha256:bfaeaf77f180f62f3994ccdc2be80dd2ef7f4d25ffc8c9497dc51e6cff69711e 2023-01-11T23:03:02.6308401Z deleted: sha256:02f5a9d8be5a1bdd5a350d4c47147fd3dd46bfcedc7637f53a8a692720381fc2 2023-01-11T23:03:02.6308831Z deleted: sha256:e22cc66e4fd2e491fb4ae8194c35d6b1789f9f5d01e1dfbedf1c266c3a1537de 2023-01-11T23:03:02.6309297Z deleted: sha256:1536d02ae84ab410916541408cf2935f122735cc5d128324f6f82fbbeb913e80 2023-01-11T23:03:02.6309726Z deleted: sha256:a8374bc83a4bf3a838aaf8ee71b4a8281ac4eef801473b007f88b2f0efadc6c6 2023-01-11T23:03:02.6310150Z deleted: sha256:9381921b1e612b2d23517d24662a39f00be43efe2412e440ef41b487a48cb389 2023-01-11T23:03:02.6310555Z deleted: sha256:ae063d6cbeae3688a5a7bb8694d431dbb9792bfb7ff2908d3b25842f9586fc8e 2023-01-11T23:03:02.6310991Z deleted: sha256:595397ba048a351c7b09c25b5eba4cbb916c2db40dd80bde4d95c2b51b766045 2023-01-11T23:03:02.6311439Z deleted: sha256:7a2da4ffb8ea2b858fcdfc92f6e640dbb6a083b39dfa5e7aa87cd1296f9314e1 2023-01-11T23:03:02.6311905Z deleted: sha256:eddd66bfdbb5913f133ffda8d967ee0235f9f121434112ea8da4cfdf5f9ebbf4 2023-01-11T23:03:02.6312328Z deleted: sha256:c9392d92ee837d52b35b41e4de67d213886d844cecd9e769d9284dc21070aee8 2023-01-11T23:03:02.6312770Z deleted: sha256:fba16b8beafe9efa854d93f0e92718750ec97ced755ace0f6f51bbd5d1964f91 2023-01-11T23:03:02.6313199Z deleted: sha256:a09592e27d2e6896b9029f31e269787c92761fd19a48528851c48d85221cd4bc 2023-01-11T23:03:02.6313578Z deleted: sha256:8b2e3a8416af60625ccb9a8562891c1b9e85c2ee05f103b190ad5040313bf1f1 2023-01-11T23:03:02.6314004Z deleted: sha256:135caddb443044601f433d421e5c0f5d8ab02dff69cf2df9024a8ecb97c8948c 2023-01-11T23:03:02.6314407Z deleted: sha256:112b60db47585e101175390e102a66908f0f1175510c5e5d5f10b7e4e0c9769b 2023-01-11T23:03:02.6314807Z deleted: sha256:d9e5dd4e760b68190c010a9042c842afb0bcf3d4477a334d0e9c2d9c302ecb3c 2023-01-11T23:03:02.6315217Z deleted: sha256:7d37673dfb91518db6e68e9637f3db142a4eaccb9f548ab99d14ac52b3672325 2023-01-11T23:03:02.6315634Z deleted: sha256:12784d32c23b63941ebc8adf713d7167158c38c771d3b5f94506e036b6273dbd 2023-01-11T23:03:02.6316047Z deleted: sha256:395db4c8bde7e390accfa81ff84376004b78566423bd0a3bd7559e12e66759bc 2023-01-11T23:03:02.6316454Z deleted: sha256:a7aa04ee64333d427277f47b8e07dee6cc566b132a18ee166c305c890d4fd3bb 2023-01-11T23:03:02.6316866Z deleted: sha256:10a0e8d138614f29a13aecd15175039c7f86ca04fc588b439f56284b3c0e292a 2023-01-11T23:03:02.6317289Z deleted: sha256:68d97ecc8d2ec5f756b4a1b7e7451a413ed19f3d7c3ce52c81b30dc00fd96185 2023-01-11T23:03:02.6317714Z deleted: sha256:e833f7b95e0efaaf775293760365243991395e796d587deffe588e50fe7a9f1e 2023-01-11T23:03:02.6318104Z deleted: sha256:cbcd1e7502e949614d05d5d0a34316fd62665c3e030b3656d9c488cfee1eca34 2023-01-11T23:03:02.6318510Z deleted: sha256:a90ce4a75d9d408df6349607788566125c2662c603dc8f84c767b2256273ff12 2023-01-11T23:03:02.6318924Z deleted: sha256:bdee06da46fd67ff14bbc7286b40ef15174551b6452ca633be8576376a3dbddb 2023-01-11T23:03:02.6319333Z deleted: sha256:2cafe83ace87d1a83f30f8d458001f3a315a606d5251c20543cbd6604499aa73 2023-01-11T23:03:02.6319760Z deleted: sha256:cc1f7fb1208e7b05b48d3b3b2648946adaa723b10df2574c7db87ed9112b1510 2023-01-11T23:03:02.6320186Z deleted: sha256:9027ec66ecfd6490ecf157cdf665428dad438f54565d9d85263211da30e6684d 2023-01-11T23:03:02.6320614Z deleted: sha256:a27541cda46a9ede5932b1a1807360e2f6ada5bb0c30bfa3aa953d59dcc1bc5c 2023-01-11T23:03:02.6321124Z deleted: sha256:7b927de6f9fdeb74acee54e7654d04e2614a112cafb477b2433db34dcc7ebe28 2023-01-11T23:03:02.6321554Z deleted: sha256:8298c8753925ad5124b0611c4101e92ec1f877252f8320f5503ebf3e4e7e1314 2023-01-11T23:03:02.6322015Z deleted: sha256:0b741a2d83533cb3b47b912506d486fd477d8c8a1c520a0f9d7d62edfc55487d 2023-01-11T23:03:02.6322420Z deleted: sha256:9412a0a6fd6057a9939fed5beadf64568de1d230c0325a628e396f5b76444bbb 2023-01-11T23:03:02.6322867Z deleted: sha256:0a294dad6a5df3fab407cd28b31eadaf9f65a5bdbf8c7a0db349d3401b537cef 2023-01-11T23:03:02.6323311Z deleted: sha256:b42ee7d4f5715886c70d5cfef4724e889f773a20741ebc4ed9c1771eb09ad634 2023-01-11T23:03:02.6323731Z deleted: sha256:a68106c7b0e03f1e0fa9ad405ce403e2931174825bdcd8e259522ef40ec8a617 2023-01-11T23:03:02.6324132Z deleted: sha256:ad403ee05a64f55c765e665bd3f25a71650857123282dc1bad2af81f5665cda5 2023-01-11T23:03:02.6324547Z deleted: sha256:eb731dc1382c6cc80193d0454f740fc55f441c32f08cd0ce1b8784e7840e53df 2023-01-11T23:03:02.6324958Z deleted: sha256:45bbe3d22998589317c7f6c4dd591475423bb37ca9b922529c5878653483b18d 2023-01-11T23:03:02.6325184Z 2023-01-11T23:03:02.6416465Z Total reclaimed space: 21.38GB 2023-01-11T23:03:02.6480130Z Post job cleanup. 2023-01-11T23:03:02.6521793Z Post job cleanup. 2023-01-11T23:03:02.8049850Z [command]/usr/bin/git version 2023-01-11T23:03:02.8108280Z git version 2.38.1 2023-01-11T23:03:02.8174806Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/26572a8c-e859-4f99-a4e0-667d7f5721c8' before making global git config changes 2023-01-11T23:03:02.8176193Z Adding repository directory to the temporary git global config as a safe directory 2023-01-11T23:03:02.8181618Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2023-01-11T23:03:02.8233285Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2023-01-11T23:03:02.8271551Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || : 2023-01-11T23:03:02.8599769Z Entering 'android/libs/fbjni' 2023-01-11T23:03:02.8644263Z Entering 'third_party/FP16' 2023-01-11T23:03:02.8687057Z Entering 'third_party/FXdiv' 2023-01-11T23:03:02.8729525Z Entering 'third_party/NNPACK' 2023-01-11T23:03:02.8772938Z Entering 'third_party/QNNPACK' 2023-01-11T23:03:02.8814899Z Entering 'third_party/VulkanMemoryAllocator' 2023-01-11T23:03:02.8857827Z Entering 'third_party/XNNPACK' 2023-01-11T23:03:02.8911871Z Entering 'third_party/benchmark' 2023-01-11T23:03:02.8953371Z Entering 'third_party/cpuinfo' 2023-01-11T23:03:02.8995445Z Entering 'third_party/cub' 2023-01-11T23:03:02.9037301Z Entering 'third_party/cudnn_frontend' 2023-01-11T23:03:02.9085792Z Entering 'third_party/cutlass' 2023-01-11T23:03:02.9137945Z Entering 'third_party/eigen' 2023-01-11T23:03:02.9182713Z Entering 'third_party/fbgemm' 2023-01-11T23:03:02.9224649Z Entering 'third_party/fbgemm/third_party/asmjit' 2023-01-11T23:03:02.9265559Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2023-01-11T23:03:02.9308480Z Entering 'third_party/fbgemm/third_party/googletest' 2023-01-11T23:03:02.9349459Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2023-01-11T23:03:02.9392594Z Entering 'third_party/flatbuffers' 2023-01-11T23:03:02.9436188Z Entering 'third_party/fmt' 2023-01-11T23:03:02.9477673Z Entering 'third_party/foxi' 2023-01-11T23:03:02.9519364Z Entering 'third_party/gemmlowp/gemmlowp' 2023-01-11T23:03:02.9603860Z Entering 'third_party/gloo' 2023-01-11T23:03:02.9646919Z Entering 'third_party/googletest' 2023-01-11T23:03:02.9689671Z Entering 'third_party/ideep' 2023-01-11T23:03:02.9735159Z Entering 'third_party/ideep/mkl-dnn' 2023-01-11T23:03:02.9779969Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2023-01-11T23:03:02.9830946Z Entering 'third_party/ios-cmake' 2023-01-11T23:03:02.9874224Z Entering 'third_party/ittapi' 2023-01-11T23:03:02.9917357Z Entering 'third_party/kineto' 2023-01-11T23:03:02.9959272Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2023-01-11T23:03:03.0001557Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2023-01-11T23:03:03.0045017Z Entering 'third_party/nccl/nccl' 2023-01-11T23:03:03.0087735Z Entering 'third_party/neon2sse' 2023-01-11T23:03:03.0131240Z Entering 'third_party/nlohmann' 2023-01-11T23:03:03.0175998Z Entering 'third_party/onnx' 2023-01-11T23:03:03.0232530Z Entering 'third_party/onnx/third_party/benchmark' 2023-01-11T23:03:03.0276007Z Entering 'third_party/onnx/third_party/pybind11' 2023-01-11T23:03:03.0322095Z Entering 'third_party/onnx-tensorrt' 2023-01-11T23:03:03.0363203Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2023-01-11T23:03:03.0409788Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2023-01-11T23:03:03.0452674Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2023-01-11T23:03:03.0495084Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2023-01-11T23:03:03.0542945Z Entering 'third_party/pocketfft' 2023-01-11T23:03:03.0585016Z Entering 'third_party/protobuf' 2023-01-11T23:03:03.0630896Z Entering 'third_party/protobuf/third_party/benchmark' 2023-01-11T23:03:03.0673963Z Entering 'third_party/protobuf/third_party/googletest' 2023-01-11T23:03:03.0718342Z Entering 'third_party/psimd' 2023-01-11T23:03:03.0759553Z Entering 'third_party/pthreadpool' 2023-01-11T23:03:03.0802377Z Entering 'third_party/pybind11' 2023-01-11T23:03:03.0845233Z Entering 'third_party/python-enum' 2023-01-11T23:03:03.0887787Z Entering 'third_party/python-peachpy' 2023-01-11T23:03:03.0929730Z Entering 'third_party/python-six' 2023-01-11T23:03:03.0971852Z Entering 'third_party/sleef' 2023-01-11T23:03:03.1015506Z Entering 'third_party/tbb' 2023-01-11T23:03:03.1059872Z Entering 'third_party/tensorpipe' 2023-01-11T23:03:03.1102866Z Entering 'third_party/tensorpipe/third_party/googletest' 2023-01-11T23:03:03.1144807Z Entering 'third_party/tensorpipe/third_party/libnop' 2023-01-11T23:03:03.1186073Z Entering 'third_party/tensorpipe/third_party/libuv' 2023-01-11T23:03:03.1227400Z Entering 'third_party/tensorpipe/third_party/pybind11' 2023-01-11T23:03:03.1268187Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2023-01-11T23:03:03.1312845Z Entering 'third_party/zstd' 2023-01-11T23:03:03.1373877Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2023-01-11T23:03:03.1405687Z http.https://github.com/.extraheader 2023-01-11T23:03:03.1416370Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2023-01-11T23:03:03.1458616Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || : 2023-01-11T23:03:03.1778012Z Entering 'android/libs/fbjni' 2023-01-11T23:03:03.1802548Z http.https://github.com/.extraheader 2023-01-11T23:03:03.1835583Z Entering 'third_party/FP16' 2023-01-11T23:03:03.1861242Z http.https://github.com/.extraheader 2023-01-11T23:03:03.1895058Z Entering 'third_party/FXdiv' 2023-01-11T23:03:03.1919516Z http.https://github.com/.extraheader 2023-01-11T23:03:03.1952247Z Entering 'third_party/NNPACK' 2023-01-11T23:03:03.1977065Z http.https://github.com/.extraheader 2023-01-11T23:03:03.2010149Z Entering 'third_party/QNNPACK' 2023-01-11T23:03:03.2035110Z http.https://github.com/.extraheader 2023-01-11T23:03:03.2068495Z Entering 'third_party/VulkanMemoryAllocator' 2023-01-11T23:03:03.2093838Z http.https://github.com/.extraheader 2023-01-11T23:03:03.2125561Z Entering 'third_party/XNNPACK' 2023-01-11T23:03:03.2151476Z http.https://github.com/.extraheader 2023-01-11T23:03:03.2195335Z Entering 'third_party/benchmark' 2023-01-11T23:03:03.2220373Z http.https://github.com/.extraheader 2023-01-11T23:03:03.2259529Z Entering 'third_party/cpuinfo' 2023-01-11T23:03:03.2283985Z http.https://github.com/.extraheader 2023-01-11T23:03:03.2317417Z Entering 'third_party/cub' 2023-01-11T23:03:03.2342391Z http.https://github.com/.extraheader 2023-01-11T23:03:03.2375800Z Entering 'third_party/cudnn_frontend' 2023-01-11T23:03:03.2423597Z http.https://github.com/.extraheader 2023-01-11T23:03:03.2462612Z Entering 'third_party/cutlass' 2023-01-11T23:03:03.2487242Z http.https://github.com/.extraheader 2023-01-11T23:03:03.2528393Z Entering 'third_party/eigen' 2023-01-11T23:03:03.2553521Z http.https://github.com/.extraheader 2023-01-11T23:03:03.2587968Z Entering 'third_party/fbgemm' 2023-01-11T23:03:03.2613560Z http.https://github.com/.extraheader 2023-01-11T23:03:03.2645560Z Entering 'third_party/fbgemm/third_party/asmjit' 2023-01-11T23:03:03.2670224Z http.https://github.com/.extraheader 2023-01-11T23:03:03.2703008Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2023-01-11T23:03:03.2727076Z http.https://github.com/.extraheader 2023-01-11T23:03:03.2759849Z Entering 'third_party/fbgemm/third_party/googletest' 2023-01-11T23:03:03.2784169Z http.https://github.com/.extraheader 2023-01-11T23:03:03.2819181Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2023-01-11T23:03:03.2842793Z http.https://github.com/.extraheader 2023-01-11T23:03:03.2876656Z Entering 'third_party/flatbuffers' 2023-01-11T23:03:03.2902254Z http.https://github.com/.extraheader 2023-01-11T23:03:03.2937845Z Entering 'third_party/fmt' 2023-01-11T23:03:03.2962427Z http.https://github.com/.extraheader 2023-01-11T23:03:03.2995507Z Entering 'third_party/foxi' 2023-01-11T23:03:03.3020749Z http.https://github.com/.extraheader 2023-01-11T23:03:03.3053489Z Entering 'third_party/gemmlowp/gemmlowp' 2023-01-11T23:03:03.3078015Z http.https://github.com/.extraheader 2023-01-11T23:03:03.3110218Z Entering 'third_party/gloo' 2023-01-11T23:03:03.3135192Z http.https://github.com/.extraheader 2023-01-11T23:03:03.3168820Z Entering 'third_party/googletest' 2023-01-11T23:03:03.3193335Z http.https://github.com/.extraheader 2023-01-11T23:03:03.3225564Z Entering 'third_party/ideep' 2023-01-11T23:03:03.3249317Z http.https://github.com/.extraheader 2023-01-11T23:03:03.3281151Z Entering 'third_party/ideep/mkl-dnn' 2023-01-11T23:03:03.3305626Z http.https://github.com/.extraheader 2023-01-11T23:03:03.3340228Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2023-01-11T23:03:03.3364550Z http.https://github.com/.extraheader 2023-01-11T23:03:03.3404078Z Entering 'third_party/ios-cmake' 2023-01-11T23:03:03.3429349Z http.https://github.com/.extraheader 2023-01-11T23:03:03.3461799Z Entering 'third_party/ittapi' 2023-01-11T23:03:03.3485901Z http.https://github.com/.extraheader 2023-01-11T23:03:03.3517996Z Entering 'third_party/kineto' 2023-01-11T23:03:03.3543973Z http.https://github.com/.extraheader 2023-01-11T23:03:03.3576603Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2023-01-11T23:03:03.3600011Z http.https://github.com/.extraheader 2023-01-11T23:03:03.3632381Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2023-01-11T23:03:03.3656779Z http.https://github.com/.extraheader 2023-01-11T23:03:03.3690527Z Entering 'third_party/nccl/nccl' 2023-01-11T23:03:03.3715116Z http.https://github.com/.extraheader 2023-01-11T23:03:03.3748281Z Entering 'third_party/neon2sse' 2023-01-11T23:03:03.3772831Z http.https://github.com/.extraheader 2023-01-11T23:03:03.3804310Z Entering 'third_party/nlohmann' 2023-01-11T23:03:03.3829544Z http.https://github.com/.extraheader 2023-01-11T23:03:03.3864916Z Entering 'third_party/onnx' 2023-01-11T23:03:03.3888860Z http.https://github.com/.extraheader 2023-01-11T23:03:03.3935265Z Entering 'third_party/onnx/third_party/benchmark' 2023-01-11T23:03:03.3960112Z http.https://github.com/.extraheader 2023-01-11T23:03:03.3993287Z Entering 'third_party/onnx/third_party/pybind11' 2023-01-11T23:03:03.4019680Z http.https://github.com/.extraheader 2023-01-11T23:03:03.4054430Z Entering 'third_party/onnx-tensorrt' 2023-01-11T23:03:03.4078929Z http.https://github.com/.extraheader 2023-01-11T23:03:03.4110804Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2023-01-11T23:03:03.4135760Z http.https://github.com/.extraheader 2023-01-11T23:03:03.4173583Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2023-01-11T23:03:03.4198616Z http.https://github.com/.extraheader 2023-01-11T23:03:03.4232656Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2023-01-11T23:03:03.4258018Z http.https://github.com/.extraheader 2023-01-11T23:03:03.4291452Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2023-01-11T23:03:03.4316288Z http.https://github.com/.extraheader 2023-01-11T23:03:03.4649936Z Entering 'third_party/pocketfft' 2023-01-11T23:03:03.4675624Z http.https://github.com/.extraheader 2023-01-11T23:03:03.4707812Z Entering 'third_party/protobuf' 2023-01-11T23:03:03.4732064Z http.https://github.com/.extraheader 2023-01-11T23:03:03.4768005Z Entering 'third_party/protobuf/third_party/benchmark' 2023-01-11T23:03:03.4792997Z http.https://github.com/.extraheader 2023-01-11T23:03:03.4825447Z Entering 'third_party/protobuf/third_party/googletest' 2023-01-11T23:03:03.4849527Z http.https://github.com/.extraheader 2023-01-11T23:03:03.4884456Z Entering 'third_party/psimd' 2023-01-11T23:03:03.4910535Z http.https://github.com/.extraheader 2023-01-11T23:03:03.4943065Z Entering 'third_party/pthreadpool' 2023-01-11T23:03:03.4967725Z http.https://github.com/.extraheader 2023-01-11T23:03:03.5001010Z Entering 'third_party/pybind11' 2023-01-11T23:03:03.5025953Z http.https://github.com/.extraheader 2023-01-11T23:03:03.5058680Z Entering 'third_party/python-enum' 2023-01-11T23:03:03.5085427Z http.https://github.com/.extraheader 2023-01-11T23:03:03.5119411Z Entering 'third_party/python-peachpy' 2023-01-11T23:03:03.5145401Z http.https://github.com/.extraheader 2023-01-11T23:03:03.5178518Z Entering 'third_party/python-six' 2023-01-11T23:03:03.5203169Z http.https://github.com/.extraheader 2023-01-11T23:03:03.5235914Z Entering 'third_party/sleef' 2023-01-11T23:03:03.5261465Z http.https://github.com/.extraheader 2023-01-11T23:03:03.5295306Z Entering 'third_party/tbb' 2023-01-11T23:03:03.5320100Z http.https://github.com/.extraheader 2023-01-11T23:03:03.5355109Z Entering 'third_party/tensorpipe' 2023-01-11T23:03:03.5380940Z http.https://github.com/.extraheader 2023-01-11T23:03:03.5413828Z Entering 'third_party/tensorpipe/third_party/googletest' 2023-01-11T23:03:03.5437695Z http.https://github.com/.extraheader 2023-01-11T23:03:03.5470975Z Entering 'third_party/tensorpipe/third_party/libnop' 2023-01-11T23:03:03.5495116Z http.https://github.com/.extraheader 2023-01-11T23:03:03.5526541Z Entering 'third_party/tensorpipe/third_party/libuv' 2023-01-11T23:03:03.5551538Z http.https://github.com/.extraheader 2023-01-11T23:03:03.5584055Z Entering 'third_party/tensorpipe/third_party/pybind11' 2023-01-11T23:03:03.5607943Z http.https://github.com/.extraheader 2023-01-11T23:03:03.5639507Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2023-01-11T23:03:03.5664550Z http.https://github.com/.extraheader 2023-01-11T23:03:03.5700232Z Entering 'third_party/zstd' 2023-01-11T23:03:03.5724799Z http.https://github.com/.extraheader 2023-01-11T23:03:03.6037773Z Cleaning up orphan processes